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ABSTRACT 


In  anticipation  of  the  implementation  of  the  Directory  of  DoD-Sponsored 
R&D  Data  Bases  in  an  on-line  version  on  the  Defense  Gateway  Computer  System 
(hereafter  the  Gateway),  a  study  was  undertaken  to  identify  the  searching 
requirements  of  existing  and  potential  users.  The  terms  "user-friendly 
interface,  natural  language  front-end  processor,"  and  "expert  system"  are 
defined.  The  procedure  followed  in  conducting  the  study  is  described. 
Results  of  the  study  are  presented  along  with  a  recommendation  for  an 
interface  to  be  incorporated  into  the  Gateway  for  searching  the  on-line 
version  of  the  directory. 
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INTRODUCTION 


As  technology  advances  and  systems  and  methods  of  information  retrieval 
become  more  and  more  complex,  there  arises  a  greater  need  to  simplify  the 
searching  of  data  bases  so  that  the  users  of  the  information  can  do  their  own 
searching.  An  interface  can  be  described  as  "a  hardware/software  layer  that 
can  be  interposed  between  users  and  a  database  system"^  to  simplify  the 
process  of  searching  the  data  base  or  data  bases.  Its  purpose  is  to  act  as 
an  intermediary  to  assist  users  in  accessing  and  searching  heterogenous 
retrieval  systems. 

As  the  Defense  Technical  Information  Center  (DTIC)  develops  new  means  to 
access  heterogenous  data  bases  (the  Defense  Gateway  Computer  System),  the 
need  for  an  interface  also  becomes  apparent. 

The  Defense  Gateway  Computer  System  (hereafter  the  Gateway),  now  in  test 
phase,  is  being  developed  to  provide  on-line  access  to  data  bases  of  interest 
to  the  DoD  RDT&E  community.  At  the  time  this  paper  is  being  written,  the 
Gateway  can  access  the  Defense  RDT&E  On-line  System  (DROLS),  NASA/RECON, 
DOE/RECON,  DIALOG  and  other  data  bases.  The  Directory  of  DoD-Sponsored  R&D 
Data  Bases  (hereafter  the  directory)  will  be  made  available  for  searching  on 
the  Gateway,  as  will  some  of  the  data  bases  described  in  the  directory.  The 
directory,  in  on-line  version,  will  be  called  the  Data  Base  of  Data 
Bases.  The  abstract  from  DD  Form  1473  describes  the  directory  as  follows. 

The  Directory  of  DoD  R&D  Data  Bases  is  a  listing  of  DoD's  R&D  data 
bases.  Each  entry  contains  information  such  as  the  data  base 
name,  dates  of  coverage,  points  of  contact,  hardware/software 
configuration ,  and  a  description  of  the  data  base.  Agency, 
data  base,  and  subject  indices  are  provided.  The  subject  coverage 
includes  meteorology,  weapon  systems,  hazardous  materials, 
medicine,  oceanography,  antennas,  survivability,  reliability,  and 
chemistry. 
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PURPOSE  OF  STUDY 


In  anticipation  of  the  implementation  of  the  directory  in  an  on-line 
version  on  the  Gateway,  the  purpose  of  this  study  will  be  to  identify  the 
searching  requirements  of  existing  and  potential  users  of  the  directory. 
These  requirements  will  suggest  specifications  for  an  interface  to  be 
incorporated  into  the  Gateway  system  for  searching  the  Data  Base  of  Data 
Bases. 


i 
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BACKGROUND 


Interfaces  have  been  developed  to  make  searching  of  data  bases  easier  for 
the  user  of  the  system.  Several  kinds  of  searching  obstacles  may  be  overcome  by 
the  addition  of  an  appropriate  and  useful  interface,  for  example,  choosing  the 
correct  data  base,  connecting  to  a  communication  network,  connecting  and  logging 
on  to  a  data  base,  communicating  with  the  data  base  in  its  native  language  and 
understanding  the  replies  from  the  data  base.  If  information  is  sought  from 
more  than  one  data  base,  an  important  function  of  an  interface  identified  by 

Lancaster  and  Smith  is  "to  shield  users  from  the  incompatibilities  that  exist 

3 

among  various  systems  and  data  bases." 

In  his  1978  article,  Charles  Goldstein  noted  that  "different  user  classes 
are  identified  (professional  intermediary  searchers,  casual  infrequent  users, 
etc.)  but  different  classes  of  users  are  not,  by  and  large,  reflected  in  the 
actual  user  interface."^  The  users  of  a  data  base  may  be  members  of  a 
heterogenous  population  with  a  variety  of  searching  needs.  It  is  important, 
when  developing  an  interface,  to  keep  these  searching  needs  in  mind.  An 
interface  should  not  be  developed  which  is  more  sophisticated  than  the  potential 
user  population  requires.  For  those  reasons  the  interface  which  will  be 
developed  to  search  the  on-line  version  of  the  directory  -  the  Data  Base  of  Data 
Bases  -  on  the  Gateway  should  include  features  which  have  been  identified  as 
necessary  by  the  potential  users  of  the  Data  Base  of  Data  Bases.  This  study 
will  report  the  results  obtained  from  actually  contacting  a  sample  group  of 
these  users. 

The  results  of  this  study  will  suggest  what  type  of  interface  will  be  most 
effective  for  the  Data  Base  of  Data  Bases  on  the  Gateway,  and  what  features  it 
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should  include.  These  features  will  be  compared  to  features  of  four  existing 
interfaces:  CONIT  (Conversion  for  Network  Information  Transfer),  a 

user-friendly  interface  for  searching  commercial  on-line  bibliographic  data 
bases  developed  by  Dr.  Richard  Marcus  at  the  Massachusetts  Institute  of 
Technology;  ASSIST,  a  more  sophisticated  version  of  CONIT,  which  includes  some 
expert  system  features,  also  developed  by  Dr.  Marcus;  FRED  (a  FRont  End  for  Data 
Bases),  a  natural  language  front-end  processor  developed  by  Dr.  Gabriel  Jakobson 
at  GTE;  and  CITE  (Current  Information  Transfer  in  English),  a  natural  language 
front-end  processor  with  some  expert  system  capabilities  developed  to  search  the 
MEDLINE  data  base  by  Dr.  Tamas  Doszkocs  at  the  National  Library  of  Medicine. 

A  number  of  articles  have  been  written  that  describe  these  four  interfaces 
in  considerably  greater  detail.  For  example,  Marcus^  describes  the 
philosophy,  design,  and  implementation  of  his  experimental  interface  called 
CONIT,  and  evaluates^  the  concept  and  its  effectiveness.  Crystal  and 
Jakobson ?s  article'7  defines  the  concept  and  functions  of  FRED,  the  interface 
they  developed  at  GTE.  A  natural  language  interface  to  MEDLINE,  called  CITE  by 

O  Q 

its  developers,  is  described  in  detail  by  Doszkocs  and  Rapp  .  Marcus 
describes  EXPERT,  an  earlier  version  of  ASSIST,  as  a  computer  intermediary 
system  which  simulates  an  expert  human  information  specialist. 

Artificial  intelligence  has  been  proposed  as  a  possibility  for  making 
on-line  systems  available  to  a  wider  range  of  people.  Smith^  presented 
this  idea  as  early  as  1980.  Other  authors  who  suggest  expert  systems  that  can 
be  used  as  on-line  search  intermediaries  are  DeJong^,  Obermeier  and 
Cooper1  ,  Pollitt1  ,  and  Walker  and  Janes1  .  Yaghmai  and 

Maxin^  present  a  state-of-the-art  overview  of  expert  systems,  how  they  work 
and  their  uses  in  the  library /inf ormation  science  field. 
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DEFINITIONS 


Before  a  detailed  description  of  this  study  can  begin,  a  few  terms  must 
be  defined  and  some  features  must  be  described.  The  terms  to  be  defined  are 
user  friendly  interface,  natural  language  front-end  processor,  and  expert 
system. 

User-friendly  interface  -  While  there  is  no  consensus  in  the  field,  for  the 
purposes  of  this  study  a  "user-friendly  interface"  is  defined  as  one  which 
makes  a  data  base  easy  to  use  by  any  user.  A  user-friendly  interface 
includes  features  which  allow  a  system  to  be  used  by  a  heterogenous  user 
population.  For  example  a  user-friendly  interface  may  include  a  common 
command  language,  a  series  of  menus  from  which  to  choose  actions,  or  a  help 
feature  in  which  the  system  will  explain  in  greater  detail  a  segment  of  the 
search  process  if  asked. 

An  example  of  a  user-friendly  interface  is  CONIT,  developed  by  Dr. 
Richard  Marcus  at  MIT.  CONIT  incorporates  a  common  command  language  for 
searching  different  commercial  on-line  bibliographic  data  bases  which 
normally  have  different  languages  and  protocols.  CONIT  also  provides 
extensive  instructional  dialogue  for  the  inexperienced  searcher. 


Natural  language  front-end  processor  -  A  natural  language  front-end  processor 
is  a  type  of  interface  that  can  be  inserted  into  a  data  base  system  between 
the  data  base  and  the  user.  It  can  accept  natural  English  language  queries, 
compensate  for  spelling  errors,  unknown  words  and  partial  sentences,  and 
respond  with  natural  English  language  output. 
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An  example  is  FRED,  a  natural  language  front-end  processor  developed  by 
Dr.  Gabriel  Jakobson  at  GTE.  "User  queries  and  commands  are  routed  to  FRED 
which  sets  up  appropriate  data  base  connections  and  makes  necessary  language 
translations  so  that  different  data  bases  all  have  uniform  appearance  to  the 
user.  FRED  accepts  English  language  queries  and  compensates  for  any 
input  errors. 

A  natural  language  front-end  processor  may  also  include  a  knowledge  base 
containing  the  content  and  vocabulary  of  the  specific  subject  area  it  was 
designed  to  interpret.  Thus,  it  could  "understand”  the  context  of  a  query. 

An  example  of  such  an  interface  is  CITE,  a  natural  language  front-end 
processor  developed  for  searching  the  MEDLINE  data  base  by  Dr.  Tamas  Doszkocs 
at  the  National  Library  of  Medicine.  CITE  is  based  on  the  Medical  Subject 
Headings  (MeSH)  vocabulary,  which  is  a  controlled  hierarchical  vocabulary.  A 
linkage  capability  exists  from  the  text  words  in  a  query  to  appropriate  MeSH 
headings,  thus  allowing  English  language  query  input.  Other  features  of  CITE 
include  ranked  output,  which  is  the  display  of  retrieved  documents  in  order 
by  degree  of  satisfaction  of  the  user  query,  and  relevance  feedback,  which  is 
the  provision  of  the  text  words  and  MeSH  headings  under  which  the  documents 
were  indexed.^ 

Expert  system  -  An  expert  system  is  an  application  of  artificial  intelligence 
that  can  be  used  to  solve  problems  that  would  otherwise  require  human 
intervention  to  apply  reasoning  and  experience.  An  expert  system  for 
information  retrieval  would  have  some  knowledge  of  the  task  domain — on-line 
searching  of  data  bases — and  would  perform  the  reasoning  processes  of  a 
specialist  in  the  task. 
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An  expert  system  consists  of  a  data  base,  a  rule  base  and  a  rule 
interpreter. 

1.  The  data  base  is  a  term  for  the  working  memory  where  factual 

information  is  stored. 

2.  The  rule  base  or  knowledge  base  is  a  file  of  judgemental  rules 

applicable  to  a  specified  set  of  problems;  the  rules  are  obtained 
from  human  experts.  These  rules  are  also  known  as  heuristics. 

3.  The  rule  interpreter  or  inference  engine  applies  the  rules  in  the 

rule  base  to  the  facts  stored  in  the  data  base.^ 

The  single  most  important  feature  that  characterizes  an  expert  system  is 

its  ability  to  make  decisions  and  reveal  the  logic  it  followed  in  making 

those  decisions.  Thus,  an  expert  system  could  provide  answers  to  a  query  in 

an  on-line  bibliographic  data  base  by  giving  "detailed  information  on 

1 9 

documents,  including  why  they  were  retrieved."  Such  "relevancy 

judgments"  can  also  aid  in  refining  search  strategy  by  "finding  new  good 

20 

search  terms  or  emphasizing  the  importance  of  terms  already  in  use." 

In  order  to  design  an  expert  system  for  on-line  searching,  it  would  be 
necessary  not  only  to  identify  what  a  human  expert  searcher  needs  to  know  to 
be  an  expert  searcher,  but  also  to  apply  this  expertise.  Since  no  one  has 
yet  been  successful  in  passing  this  expertise  on  to  a  machine,  no  expert 
system  for  on-line  searching  of  data  bases  exists.  But  a  few  interfaces  now 
in  development  do  possess  some  expert  system  features. 

For  example,  CITE  will  rank  retrieved  records  according  to  their 
relevancy,  and  it  will  suggest  related  terms  so  that  a  search  can  be  expanded 
to  find  other  items  that  contain  terminology  similar  to  the  selected 
citations.  ASSIST,  a  more  sophisticated  version  of  CONIT  also  developed  by 
Dr.  Marcus,  includes  some  features  of  an  expert  system.  ASSIST  questions  the 
user  to  refine  and  reformulate  a  search  strategy,  and  it  displays  the  process 
it  followed  to  obtain  the  search  results. 
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METHODOLOGY 


The  plan  for  this  study  was  to  contact  a  sample  group  of  people  who  were 
familiar  with  the  directory  to  ask  them  how  they  used  the  print  directory, 
and  try  to  determine  their  searching  requirements  for  an  on-line  version  of 
the  directory. 

A  questionnaire  was  used  to  gather  this  information.  This  instrument 
was  chosen  to  define  and  standardize  the  information  that  would  be  gathered. 
This  standardization  served  to  increase  reliability  and  facilitate  analysis 
of  the  results.  It  was  decided  that  telephone  interviews  would  be  used  to 
gather  participant  responses  rather  than  mailing  out  the  questionnaires, 
because  the  results  needed  to  be  gathered  within  a  limited  time  frame.  The 
advantages  of  the  telephone  interview  are  rapid  completion  with  a  high 
response  rate. 

Due  to  the  length  and  complexity  of  the  questionnaire  and  the  short  time 
frame  in  which  the  study  was  to  be  completed,  it  was  decided  that  a  copy  of 
the  questionnaire  would  be  sent  to  each  participant  on  the  list  along  with  a 
memo  explaining  the  purpose  of  the  study.  The  participants  were  then 
contacted  by  telephone  with  the  questionnaire  serving  as  a  script.  This 
method  of  direct  contact  allowed  for  a  standardized  questioning  of  each 
participant.  The  participants  had  the  opportunity  to  be  prepared  ahead  of 
time  and  had  time  to  think  about  their  answers.  If  the  participants  had  any 
questions,  they  could  easily  be  answered  by  the  interviewer. 
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PROCEDURE 


SELECTION  OF  POTENTIAL  PARTICIPANTS 

A  list  of  potential  study  participants  was  compiled.  This  list  appears 
in  Appendix  B.  The  list  was  compiled  in  the  following  way. 

A  total  of  39  people  were  invited  to  participate  in  the  study.  Twelve 
people  who  are  presently  Gateway  users  were  chosen  because  of  their 
familiarity  with  the  Gateway.  They  were  contacted  and  a  copy  of  the 
directory  was  sent  to  them  if  they  did  not  yet  have  a  copy.  Eight 
individuals  from  organizations  with  listings  in  the  directory  were  chosen 
because  the  contact  person  was  known  to  have  an  interest  in  the  Gateway. 

They  had  been  sent  copies  of  the  directory  as  a  result  of  being  listed  in  it. 
Ten  individuals  from  organizations  which  participate  in  the  Shared 
Bibliographic  Input  Network  (SBIN)  were  chosen  because  of  a  demonstrated 
interest  in  new  technologies.  The  contact  persons  of  these  ten  organizations 
were  also  on  a  list  of  those  who  had  requested  and  been  sent  a  copy  of  the 
directory.  Finally  nine  individuals  from  miscellaneous  organizations  were 
chosen  to  question  because  the  contact  person  was  known  to  have  an  interest 
in  the  Gateway  and/or  the  directory.  These  individuals  were  also  on  the  list 
as  having  requested  and  received  a  copy  of  the  directory. 
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PROCEDURE 


CONSTRUCTION  OF  THE  QUESTIONNAIRE 


The  questionnaire  was  compiled  by  a  committee  consisting  of  the  author,  and 
committee  members,  Carol  Jacobson,  Marjorie  Powell,  Marcia  Hanna,  and  Gladys 
Cotter.  The  final  version  of  the  questionnaire  which  was  used  in  this  study 
appears  in  Appendix  A. 

Questions  number  1—7  were  designed  to  inform  the  prospective  participant  of 
the  purpose  of  the  telephone  call  and  to  determine  if  and  when  the  person  was 
willing  to  participate. 

Question  number  8  is  self-explanatory.  Question  number  9  served  to 
differentiate  those  participants  who  were  librarians  and  probably  did  on-line 
searching  of  data  bases  for  other  people.  Questions  number  10-12  were  answered 
only  by  this  group.  Those  participants  who  answered  No  to  question  number  9 
were  assumed  to  be  end  users  who  probably  did  on-line  searching  of  data  bases 
for  their  own  purposes,  or  did  not  do  any  on-line  searching  at  all.  Questions 
number  13-17  were  answered  only  by  this  group. 

Question  number  18  identified  those  participants  who  presently  used  the 
directory;  question  number  20  identified  those  participants  who  would  be  using 
the  directory  in  the  future.  The  description  of  the  directory  in  question 
number  18  was  quoted  directly  from  the  abstract  of  the  DD  Form  1473  for  the 
directory. 

Question  number  19  identified  the  indices  most  consulted  by  the  present 
users  of  the  directory. 

Question  number  21  indicated  whether  participants  felt  they  would  still 
need  the  directory  in  hard-copy  if  it  were  available  on-line. 
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Question  number  22  provided  the  number  of  people  who  would  be  using  the 
on-line  version  of  the  directory  when  it  became  available. 

The  information  in  the  description  of  the  Gateway  was  taken  from  the 
Research  and  Development  Project  Summaries,  October,  1984,  issued  by  the  Defense 
Technical  Information  Center,  Office  of  Information  Systems  and  Technology. 

Question  number  23-24  identified  those  who  would  be  doing  the  actual 
searching  of  the  on-line  version  of  the  directory. 

Question  number  25  was  directional.  Question  number  26  was  designed  to 
ascertain  whether  a  substantial  number  of  end  users  might  do  their  own  searching 
of  the  on-line  version  of  the  directory  if  it  was  easy  to  use. 

Question  number  27  indicated  how  participants  were  likely  to  use  the 
on-line  version  of  the  directory  and  what  kinds  of  information  they  would  be 
using  the  directory  for.  The  fields  in  the  directory  in  questions  number  27  and 
30(b)  were  quoted  from  the  Guide  to  Data  Base  Entries  (page  v)  in  the  directory. 
Any  additional  fields  not  included  in  the  present  version  of  the  directory  which 
participants  felt  would  be  useful  were  identified  by  questions  number  28  and 
29. 

Question  number  30  identified  the  fields  which  participants  felt  they  did 
not  need  to  see  displayed.  Any  additional  fields  not  included  in  the  present 
version  of  the  directory  which  participants  felt  they  would  like  to  see 
displayed  were  identified  by  questions  number  31  and  32. 

The  list  of  features  (questions  number  33-60)  which  could  be  included  in 
the  on-line  version  of  the  directory  was  gathered  from  articles  describing 
ASSIST,  CITE,  CONIT  and  FRED.  Other  features  which  the  committee  felt  would  be 
useful  in  an  interface  for  the  directory  were  added  to  the  list  drawn  from  the 
journal  articles. 
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Additional  searching  requirements  not  mentioned  in  questions  number  33-60 
which  the  participants  felt  they  would  need  for  searching  an  on-line  version  of 
the  directory  were  identified  by  questions  number  61  and  62. 

The  questionnaire  was  designed  to  gather  more  information  than  was 
necessary  for  this  study.  Although  all  the  results  are  presented  in  this 
report,  only  the  results  to  questions  number  34-60  are  analyzed  in  order  to  make 
a  recommendation.  The  purpose  of  this  study  was  to  define  user  searching 
requirements  for  the  Directory  of  DoD-Sponsored  R&D  Data  Bases,  however,  some  of 
the  data  bases  described  in  the  directory  will  also  be  available  on-line 
eventually,  and  accessible  through  the  Gateway.  An  interface  will  then  be 
necessary  for  cross-searching  these  data  bases.  At  that  point,  information  on 
searching  requirements  would  again  have  to  be  gathered,  probably  from  the  same 
sample  group  of  participants.  The  additional  information  gathered  in  this  study 
and  presented  but  not  analyzed  in  this  report  will  be  used  at  a  later  date. 

A  draft  version  of  the  questionnaire  was  pretested  on  3-4  January  1985  with 
two  DTIC  employees  under  conditions  similar  to  those  in  which  the  study  would  be 
conducted.  Suggestions  for  changes  were  made  by  the  respondents  and  these 
changes  were  incorporated  by  the  author  into  the  final  version  of  the 
questionnaire.  Copies  of  the  questionnaire  were  mailed  to  participants  on  7 
January  1985.  The  telephone  interviews  were  begun  on  11  January  1985  and 
concluded  on  25  January  1985. 
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PROBLEMS  ENCOUNTERED 


Some  problems  encountered  with  specific  questions  in  the  questionnaire 
are  as  follows. 

Question  number  8,  "Approximate  number  of  people  in  organization",  was 
meant  to  indicate,  in  the  case  of  librarians,  the  number  of  people  working  in 
their  library.  Unfortunately  some  participants  misunderstood,  and  answered 
question  number  8  with  the  number  of  people  in  their  building,  organization, 
base,  etc.  The  data  collection  was  partially  completed  before  this 
misunderstanding  was  realized,  so  the  responses  for  this  question  are  not 
included  in  the  results. 

Question  number  31  repeats  question  number  28.  In  order  to  fit  into  the 
questionnaire  better,  question  number  31  should  read,  "Are  there  any 
additional  fields  which  you  would  like  to  display  in  the  on-line  version?" 

Question  number  33  caused  some  confusion.  Some  of  the  participants 
thought  that  the  interface  which  could  include  the  features  indicated  would 
be  used  to  search  only  the  on-line  version  of  the  directory.  The  paragraph 
in  the  questionnaire  describing  the  Gateway  (between  questions  number  22  and 
23)  states  that  Some  of  the  data  bases  described  in  the  directory  will  be 
accessible  through  the  Gateway."  Question  number  33  should  probably  have 
said  something  to  the  effect  that  the  features  would  be  included  in  an 
interface  that  would  be  used  to  search  the  data  bases  on  the  Gateway  as  well 
as  the  directory.  The  confusion,  when  it  occurred,  was  noticed  and  corrected 
before  the  questionnaire  was  completed;  it  is  not  likely  that  the  results 
were  affected. 
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RESULTS 


Of  the  39  people  asked  to  participate  in  this  study,  32  (82%)  completed 
the  questionnaire  (see  Appendix  A).  From  this  group,  23  people  (71.8%) 
identified  themselves  as  a  member  or  supervisor  of  the  library  staff  in 
question  number  9. 

Results  of  questions  number  9-12  pertaining  to  this  group  of  participants 
are  shown  in  Tables  1-3. 

The  responses  to  question  number  10  are  shown  in  Table  1.  The  number  of 
individuals  doing  on-line  searching  of  data  bases  is  divided  into  categories 
(i.e.,  1—3,  4—6).  The  number  of  responses  is  recorded  under  these  categories. 


TABLE  1 

INDIVIDUALS  DOING  ON-LINE  SEARCHING  OF  DATA  BASES 


Number 

of 

individuals 

1-3 

4-6 

7-9 

over  9 

Number 

of 

responses 

8 

10 

4 

1 

The  responses  to  question  number  11  are  shown  in  Table  2.  The  most 
^^-"^Huontly  searched  data  bases  are  listed  in  the  left  column.  The  number  of 
participants  searching  these  data  bases  is  recorded  in  the  column  under 
Responses.  The  percentage  of  the  total  group  (23  people)  indicating  that  they 
searched  that  particular  data  base  is  included  in  the  right  column. 
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TABLE  2 


DATA  BASES  SEARCHED  MOST  FREQUENTLY 


Data  Bases 

Responses 

°/ 

/ 0 

DROLS 

23 

100% 

DIALOG 

17 

74% 

OCLC 

12 

52% 

BRS 

10 

43.4% 

NASA/RECON 

8 

34.8% 

ORBIT 

7 

30% 

NOTE.  Other  data  bases  searched  included:  in-house  data  bases,  intelligence 
data  bases,  Chemical  Abstracts,  DoE/RECON,  International  Data  Bases,  NEXIS, 
FAXON,  and  MEDLINE. 


The  responses  to  question  number  12  are  shown  in  Table  3.  The  hardware 
available  to  the  participants  of  this  group  are  listed  in  the  left  column.  The 
number  of  No  responses  is  listed  in  the  next  column.  The  number  of  Yes 
responses  to  question  number  12  is  divided  into  categories  by  how  many  of  each 
type  of  hardware  was  available  to  the  participant. 


TABLE  3 
HARDWARE  USED 


No 

1 

Yes 

2 

3 

over  3 

Dedicated  terminals 

3 

9 

5 

3 

3 

Dial-up  terminals 

2 

11 

4 

1 

4 

Communicating  microcomputers 

16 

4 

1 

0 

3 

Dedicated  microcomputers 

20 

1 

0 

0 

2 
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Of  the  32  people  who  completed  the  questionnaire,  9  (28%)  responded  with  a 
No  to  question  number  9,  and  were  thus  directed  to  question  number  13.  Results 
of  questions  number  13-17  pertaining  to  this  group  of  participants  only  are 
shown  in  Tables  4-5. 

The  responses  to  question  number  13  are  shown  in  Table  4.  Participants 
described  their  work  by  more  than  one  of  the  terms  in  most  cases. 


TABLE  4 

END  USER  OCCUPATION 


No.  Responses 


Planner 

Marketer 

Scientist  or  Engineer 


6 

3 

5 


Note.  —  Other  occupations  included:  operations  manager,  program  manager, 
economist,  management,  programmer,  consultant,  technical  information 
specialist. 


In  response  to  question  number  14,  seven  menbers  of  this  sample  group  of 
nine  (77.7%)  did  their  own  on-line  searching. 

In  response  to  question  number  15,  four  participants  searched  DTICTs 
on-line  data  bases;  two  participants  searched  DIALOG.  Other  data  bases 
searched  included:  LEXIS/NEXIS,  BRS,  CIRC,  Defense  Manpower  Data  Center, 
Defense  Resources,  Inc.,  NASA/RECON,  Aerospace  Daily,  Dow  Jones,  robotics  data 
bases,  management  data  bases,  standards  data  bases,  and  directives  data  bases. 

The  responses  to  question  number  16  are  shown  in  Table  5.  The  hardware 
available  to  the  participants  of  this  group  are  listed  in  the  left  column. 
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The  number  of  No  responses  are  listed  in  the  next  column.  The  number  of  Yes 
responses  to  question  number  16  are  divided  into  categories  by  how  many  of  each 
type  of  hardware  was  available  to  the  participant. 


TABLE  5 
HARDWARE  USED 


No 

1 

Yes 

2 

3 

over  3 

Dedicated  terminals  5 

3 

0 

0 

0 

Dial-up  terminals  4 

12 

0 

1 

1 

Communicating  microcomputers  3 

3 

0 

1 

1 

Dedicated  microcomputers  6 

1 

0 

0 

1 

No  responses  to  question  number  17  were  recorded. 

The  responses  to  questions  number  18-62  presented  here  reflect  the  group 
of  participants  as  a  whole  (32  people). 

In  response  to  question  number  18,  13  participants  (40.6%)  stated  that 
they  presently  used  the  directory. 

For  those  who  used  the  directory,  in  response  to  question  number  19, 
indices  used  are  shown  in  Table  6.  Participants  indicated  that  they  used  more 
than  one  index  in  most  cases.  The  indices  are  listed  in  the  left  column.  The 
number  of  participants  using  each  index  is  recorded  in  the  column  under 
Responses*  The  percentage  of  the  total  group  (13  people)  Indicating  that  they 
used  that  particular  index  is  included  in  the  right  column. 
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TABLE  6 


INDICES  USED 


Responses 

% 

Data  Base  Index 

12 

92.3% 

Organization  Index 

11 

84.6% 

Subject  Index 

13 

100% 

In  response  to  question  number  20,  all  32  participants  expected  to  use 
the  directory  in  the  future. 

In  response  to  question  number  21,  25  participants  (78%)  replied  that 
they  would  still  need  the  hard  copy  version  of  the  directory  if  it  were 
available  on-line. 

In  response  to  question  number  22,  30  participants  (93.7%)  replied  that 
their  organization  would  use  the  directory  if  it  were  available  on-line  and 
accessible  through  dial-up  terminals. 

In  response  to  question  number  23,  28  participants  (87.5%)  replied  that 
they  would  be  the  ones  searching  the  directory  on-line  on  the  Gateway.  Of 
the  group  of  participants  who  replied  No  to  question  number  23,  three 
participants  would  want  someone  to  search  the  directory  for  them.  One 
participant  replied  that  he  would  rather  use  the  directory  in  paper  form,  but 
would  use  the  Gateway  to  search  the  data  bases  themselves  if  they  were 
available  on-line. 

Those  participants  who  identified  themselves  as  members  or  supervisors 
of  a  library  staff  (see  question  number  9)  were  asked  to  respond  to  question 
number  26.  Twelve  participants  (50%)  replied  that  they  thought  their  library 
users  and/or  researchers  would  use  the  on-line  version  of  the  directory  if  it 
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were  easy  to  search.  Ten  participants  (41.6%)  replied  No  to  question  number 
26,  and  two  participants  (8.3%)  replied  that  they  were  not  sure.  Those 
participants  who  replied  that  their  library  users  and/or  researchers  would 
not  do  their  own  searching  of  the  directory  gave  the  following  reasons: 

-Their  library  was  not  set  up  to  handle  end-user  searching. 

-The  library  could  afford  only  a  limited  number  of  terminals  and  there 
were  not  enough  for  the  public. 

-They  believed  that  their  library  users/researchers  would  rather  have 
someone  else  do  their  searches. 

The  responses  to  question  number  27  are  shown  in  Table  7.  One 
participant  chose  not  to  respond  to  question  number  27.  Fields  in  the 
directory  are  listed  in  the  left  column.  The  number  of  participants  who 
indicated  that  they  would  not  search  a  particular  field  is  recorded  in  the 
middle  column  next  to  the  field  along  with  the  percentage  of  the  total  group 
responding  (31  people).  The  number  of  participants  who  indicated  that  they 
would  search  a  particular  field  is  recorded  in  the  column  to  the  right  of  the 
field  along  with  the  percentage  of  the  total  group  responding  (31  people). 


TABLE  7 

FIELDS  IN  THE  DIRECTORY  MOST  LIKELY  TO  BE  SEARCHED 


No 

% 

Yes 

% 

Data  Base  Name 

4 

13% 

27 

87% 

Acronym 

3 

9.7% 

28 

90% 

Update  frequency 

15 

48% 

16 

51.6% 

Beginning  date 

16 

51.6% 

15 

48% 

Ending  date 

14 

45% 

17 

54.8% 

Size 

18 

58% 

13 

41.9% 

Data  Base  Producer  Name 

4 

13% 

27 

87% 
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TABLE  7 — Continued 


Data  Base  Producer  Address 
Data  Base  Producer  Contact 
Data  Base  Distributor  Name 
Data  Base  Distributor  Address 
Data  Base  Distributor  Contact 
Data  Base  Generator  Name 
Data  Base  Generator  Address 
Data  Base  Generator  Contact 
Availability 
Descriptors 
Data  Base  Type 
Code  Character  Set 
Density 

Number  of  Tracks 
Labeled 

Programming  language 
Computer 
Storage  Media 
Input  Media 
Output  Media 
Documentation 

Classification  Restrictions 
Abstract 


In  response  to  question  number  28,  six  participants  indicated  that  there 
are  additional  fields  they  would  like  to  see  included  in  the  on-line  version. 
These  fields  included: 

-A  sampling  of  the  data. 

-Other  users  of  the  data  base. 

-Cost  to  purchase  or  search  the  data  base. 

-Availability  of  the  data  base  for  purchase  or  lease. 

-Operating  system  used  in  the  data  base. 

-Software  used  in  the  data  base. 

-Inclusion  of  foreign  source  material,  and  from  which  countries. 
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In  response  to  question  number  30(a),  ten  participants  (32.2%)  replied 
that  there  are  fields  in  the  directory  that  they  would  not  want  to  display. 
The  responses  to  question  number  30(b)  are  shown  in  Table  8.  The  fields  that 
these  participants  did  not  want  to  display  are  listed  in  the  left  column. 
Participants  indicated  more  than  one  field  that  they  did  not  want  to  display 
in  most  cases.  The  number  of  responses  for  each  field  is  recorded  on  the 
right  under  Responses. 


TABLE  8 

FIELDS  PARTICIPANTS  DID  NOT  WANT  TO  DISPLAY 


Field  Responses 

Density  8 

No.  of  Tracks  8 

Labeled  8 

Code  Character  Set  7 

Programming  Language  6 

Computer  6 

Storage  Media  6 

Input  Media  6 

Output  Media  6 

Documentation  3 

Data  Base  Type  2 

Data  Base  Dist.  Address  2 

Data  Base  Dist.  Contact  2 

Data  Base  Prod.  Address  1 

Data  Base  Prod.  Contact  1 

Data  Base  Dist.  Name  1 

Data  Base  Gener.  Name  1 

Data  Base  Gener.  Address  1 

Data  Base  Gener.  Contact  1 


In  response  to  question  number  31,  four  participants  indicated  that 
there  are  additional  fields  they  would  like  to  display.  These  fields 
included: 
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-More  descriptors. 

-Working  paragraph  on  purpose  of  data  base. 

-Operating  systems  used  in  the  data  base. 

-Software  used  in  the  data  base. 

-Cost. 

—Availability  of  the  data  base  for  purchase  or  lease. 

—Descriptor  terms  other  than  DRIT  terms  if  the  vocabulary  used  is 
controlled. 

One  participant  commented  that  he  would  like  a  choice  of  fields  that  can 
be  displayed  each  time  the  directory  is  searched. 

The  responses  to  questions  number  34-60  are  shown  in  the  form  of  stacked 

bar  graphs  on  the  pages  that  follow  (Figures  1-27).  For  each  graph,  the  Y 

axis  represents  the  number  of  responses  for  that  feature.  The  X  axis 

represents  each  option  in  the  scale  on  which  participants  were  asked  to  rank 

the  features.  On  the  scale  of  1  through  5,  a  1  meant  the  feature  was  not 

useful,  a  2  meant  the  feature  was  somewhat  useful,  a  3  meant  the  feature  was 

neither  not  useful  nor  essential,  in  other  words,  the  participant  was 

indifferent,  a  4  meant  the  feature  was  very  useful,  and  a  5  meant  the  feature 

was  essential.  Intermediary  responses  are  represented  by  the  cross-hatched 

area;  end  user  responses  are  represented  by  the  clear  area  on  each  bar 
2 1 

graph. 

Histograms  generated  by  the  SPSS  Batch  System  illustrating  responses  to 
questions  number  34-60  are  included  in  Appendix  C.  Cross-tabulations 
generated  by  the  SPSS  Batch  System  for  these  questions  are  included  in 
Appendix  D. 
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In  response  to  question  number  61,  eight  participants  indicated  that 
they  had  additional  requirements  which  they  would  like  to  see  included  in  the 
on— line  version  of  the  directory.  These  requirements  were: 

-A  menu  of  help  features. 

-An  on-line  tutor. 

-The  ability  to  reformat  the  order  in  which  items  in  the  record  are 
displayed. 

-Color  graphics. 

-Sort  procedures. 

-A  limiting  feature. 

-The  option  to  display,  order  or  print  off-line. 

-Selective  dissemination  of  information. 

-Full-text  capabilities. 

-Ability  to  access  the  directory  on  a  dedicated  rather  than  a  dial-up 
terminal. 

-A  cost  statement  at  the  end  of  each  file  searched. 

-Classified  access. 
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Figure  1.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Minimum  Amount  of  Keyboarding"  in  answer  to  ques¬ 
tion  number  3 U  of  the  questionnaire.  The  Y  axis  represents 
the  number  of  responses.  The  X  axis  represents  each  option 
in  the  scale  1-5.  Most  of  the  participants  in  both  categ¬ 
ories  ranked  this  feature  as  being  very  useful  or  essential 
to  them. 


End  User 
Responses 


Intermediary 

Responses 


No.  of  Responses 


1  2  3  4  5 

1=not  useful 
2=somewhat  useful 
3=indif  f  er ent 
U=very  useful 
5=essential 
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Figure  2.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Choice  Between  Commands  or  Menus"  in  answer  to 
question  number  35  of  the  questionnaire.  The  Y  axis  repre¬ 
sents  the  number  of  responses.  The  X  axis  represents  each 
option  in  the  scale  1—5.  Most  of  the  participants  in  both 
categories  ranked  this  feature  as  being  very  useful  or 
essential  to  them,  except  for  a  few  end  users  who  were 
indifferent,  and  one  intermediary  who  ranked  this  feature 
as  not  useful. 
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Figure  3.  Participants’  ranking  on  a  scale  of  1-5  of  the 
feature  "Command  Driven  Only"  in  answer  to  question  number 
36  of  the  questionnaire.  The  Y  axis  represents  the  number 
of  responses.  The  X  axis  represents  each  option  in  the 
scale  1-5.  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  not  useful  or  somewhat  useful 
to  them,  or  else  they  were  indifferent.  A  minority  of  the 
participants  ranked  this  feature  as  very  useful  or  essential 
to  them. 
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Figure  U«  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Menu.  Driven  Only"  in  answer  to  question  number 
37  of  the  questionnaire.  The  Y  axis  represents  the  number 
of  responses.  The  X  axis  represents  each  option  in  the 
scale  1-5.  Participants  ranked  this  feature  across  the 
scale;  no  significant  majority  of  opinion  is  apparent. 
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Figure  5*  Participants*  ranking  on  a  scale  of  1-5  of  the 
feature  "Common  Retrieval  Language"  in  answer  to  question 
number  38  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5*  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  very  useful  or  essential  to 
them.  A  few  participants  in  both  categories  were  indiff¬ 
erent  about  this  feature.  A  few  end  users  ranked  this 
feature  as  not  useful  to  them. 
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Figure  6.  Participants*  ranking  on  a  scale  of  1-5  of  the 
feature  "Accepts  User-Defined  Commands"  in  answer  to  ques¬ 
tion  number  39  of  the  questionnaire.  The  Y  axis  represents 
the  number  of  responses.  The  X  axis  represents  each  option 
in  the  scale  1-5*  Participants  ranked  this  feature  across 
the  scale;  no  significant  majority  of  opinion  is  apparent. 
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Figure  7.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "User  Chooses  Command  Language"  in  answer  to  ques¬ 
tion  number  2+0  of  the  questionnaire.  The  Y  axis  represents 
the  number  of  responses.  The  X  axis  represents  each  option 
in  the  scale  1-5.  A  majority  of  the  participants  who  were 
intermediaries  ranked  this  feature  as  being  very  useful  or 
essential  to  them.  A  majority  of  the  participants  who  were 
end  users  were  indifferent  about  this  feature. 
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Figure  8.  Participants’  ranking  on  a  scale  of  1-5  of  the 
feature  "Choose  Among  Levels  of  Expertise"  in  answer  to 
question  number  1*1  of  the  questionnaire.  The  Y  axis  re¬ 
presents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5.  A  majority  of  the  partici¬ 
pants  who  were  intermediaries  ranked  this  feature  as  being 
very  useful  to  them,  but  a  substantial  number  of  the  inter¬ 
mediaries  ranked  this  feature  as  being  essential  to  them, 
or  they  were  indifferent.  A  majority  of  the  participants 
who  were  end  users  ranked  this  feature  as  being  essential 
to  them. 
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Figure  9*  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Accepts  Natural  English  Language"  in  answer  to 
question  number  1+2  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5*  Participants  ranked  this 
feature  across  the  scale;  no  significant  majority  of  opinion 
is  apparent. 
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Figure  10.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Compensates  for  Spelling  Errors"  in  answer  to 
question  number  1+3  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5*  Participants  ranked  this 
feature  across  the  scale;  no  significant  majority  of  opinion 
is  apparent. 
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Figure  11.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Questions  to  Formulate  Search  Strategy"  in  answer 
to  question  number  44  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5*  Participants  ranked  this 
feature  across  the  scale;  no  significant  majority  of  opinion 
is  apparent. 
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Figure  12.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Suggests  Related  Terms"  in  answer  to  question 
number  kS  of  the  questionnaire.  The  Y  axis  represents 
the  number  of  responses.  The  X  axis  represents  each 
option  in  the  scale  1-5.  Most  of  the  participants  in  both 
categories  ranked  this  feature  as  being  essential  or  very- 
useful  to  them. 
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Figure  13.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Gives  Feedback  on  Search  Strategy"  in  answer  to 
question  number  I4.6  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5*  A  majority  of  the  partici¬ 
pants  who  were  intermediaries  ranked  this  feature  as  being 
very  useful  or  essential  to  them.  A  majority  of  the  part¬ 
icipants  who  were  end  users  were  indifferent  about  this 
feature  or  else  they  ranked  this  feature  as  being  somewhat 
useful  or  essential  to  them. 

No.  of  Responses 


30 

End  User 
Responses 


25 


Intermediary 

Responses 


15 


10 


5 


0 


Swot 


mjm 


wmm 


m:wM 


1=not  useful 
2=somewhat  useful 
3= indifferent 
I|.=very  useful 
5=essential 


1 


2 


3 


4 


5 


Figure  11+.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "System  or  User  Selects  Data  Base"  in  answer  to 
question  number  hi  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5.  A  majority  of  the  participants 
who  were  intermediaries  ranked  this  feature  as  being  either 
very  useful  or  essential  to  them.  A  majority  of  the  part¬ 
icipants  who  were  end  users  ranked  this  feature  as  being 
essential  to  them  or  being  very  useful  to  them  or  else  they 
were  indifferent. 
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Figure  .  Participants'  ranking  on  a  scale  of  of  the 
feature  "System  Chooses  the  Data  Base"  in  answer  to  question 
number  1+8  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-£.  A  majority  of  the  participants  in  both 
categories  ranked  this  feature  as  being  not  useful  to  them. 
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Figure  16.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Displays  Process  It  Followed"  in  answer  to 
question  number  l±9  of  the  questionnaire.  The  Y  axis  repre¬ 
sents  the  number  of  responses.  The  X  axis  represents  each 
option  in  the  scale  1-5.  A  large  number  of  the  participants 
in  both  categories  ranked  this  feature  as  being  essential 
to  them . 
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Figure  17*  Participants’  ranking  on  a  scale  of  1-5  of  the 
feature  "Ranks  Retrieved  Results  for  Relevancy"  in  answer 
to.  question  number  $0  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1-5*  Most  of  the  participants  who 
were  intermediaries  ranked  this  feature  as  being  either  very 
useful  or  essential  to  them.  A  substantial  number  of  part¬ 
icipants  who  were  intermediaries  were  indifferent  about  this 
feature.  Most  of  the  participants  who  were  end  users  ranked 
this  feature  as  being  either  not  useful  or  essential  to  them. 
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Figure  18.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Explains  User  Errors"  in  answer  to  question  num¬ 
ber  51  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option 
in  the  scale  1-5.  Most  of  the  participants  in  both  cate- 
gories  ranked  this  feature  as  being  very  useful  or  essential 
to  them.  A  substantial  number  of  participants  who  were 
intermediaries  were  indifferent  about  this  feature. 
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Figure  1 9 .  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Includes  a  Help  Feature"  in  answer  to  question 
number  52  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5-  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  very  useful  or  essential  to 
them.  A  substantial  number  of  participants  who  were 
intermediaries  were  indifferent  about  this  feature. 
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Figure  20.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Stores  Search  Strategy"  in  answer  to  question 
number  53  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5-  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  essential  to  them. 
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Figure  21.  Participants  *  ranking  on  a  scale  of  1-5  of  the 
feature  "User  Can  Define  Output  Formats"  in  answer  to 
question  number  5U  of  the  questionnaire.  The  Y  axis 
represents  the  number  of  responses.  The  X  axis  represents 
each  option  in  the  scale  1—5*  Most  of  the  participants  who 
were  intermediaries  ranked  this  feature  as  being  essential 
to  them.  Most  of  the  participants  who  were  end  users  ranked 
this  feature  as  being  either  essential  or  very  useful  to 
them. 
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Figure  22.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Stores  User-Defined  Formats"  in  answer  to  question 
number  55  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5-  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  very  useful  or  essential  to 
them. 
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Figure  23.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Has  Menu  of  Canned  Formats"  in  answer  to  question 
number  56  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5.  Most  of  the  participants  in  both  categories 
ranked  this  feature  as  being  very  useful  or  essential  to 
them. 
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Figure  2l+.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Allows  User  to  Create  Charts"  in  answer  to  question 
number  57  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5.  Participants  ranked  this  feature  across  the 
scale;  no  significant  majority  of  opinion  is  apparent. 
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Figure  25.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Allows  User  to  Create  Graphs"  in  answer  to  question 
number  58  of  the  questionnaire.  The  Y  axis  represents  the 
number  of  responses.  The  X  axis  represents  each  option  in 
the  scale  1-5.  Participants  ranked  this  feature  across  the 
scale;  no  significant  majority  of  opinion  is  apparent. 
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Figure  26.  Participants'  ranking  on  a  scale  of  1-5  of  the 
feature  "Allows  User  to  Download  Information"  in  answer  to 

sent^th  numb®r  59  °f  the  questionnaire.  The  Y  axis  repre¬ 
sents  the  number  of  responses.  Ihe  X  axis  represents  each 

option  in  the  scale  1-5.  Most  of  the  participants  in  both 
categories  ranked  this  feature  as  being  essential  to  them 


No.  of  Responses 


30 

End  User 
Responses 

I  8 


Intermediary- 

Responses 


15 


10 


5 


0 


1=not  useful 
2=somewhat  useful 
3=in different 
i+=very  useful 
5=essential 


-58- 


Figure  27.  Participants*  ranking  on  a  scale  of  1-5  of  the 
feature  "Allows  User  to  Reformat  Information"  in  answer  to 
question  number  60  of  the  questionnaire.  The  Y  axis  repre¬ 
sents  the  number  of  responses.  The  X  axis  represents  each 
option  in  the  scale  1-5.  Most  of  the  participants  in  both 
categories  ranked  this  feature  as  being  essential  or  very- 
useful  to  them. 
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DISCUSSION  OF  SIGNIFICANT  GRAPHS 


In  a  large  number  of  the  stacked  bar  graphs  (Figures  1-27), 
intermediaries’  and  end  users’  responses  appeared  to  be  similar,  for  example, 
if  a  majority  of  the  intermediaries’  responses  in  ranking  a  particular 
feature  were  in  the  positive  range  of  the  scale,  a  majority  of  the  end  users’ 
responses  in  ranking  the  same  feature  were  also  in  the  positive  range  of  the 
scale.  Some  of  the  intermediaries’  and  end  users’  responses  did  not  appear 
similar,  but  the  differences  may  not  be  significant. 

In  order  to  determine  if  there  were  any  features  with  a  significant 
difference  between  the  way  intermediaries  and  end  users  ranked  the  feature,  a 
chi-square  test  was  done  using  the  SPSS  Batch  System.  This  chi-square  lest 
calculated  the  independence  of  the  two  variables-intermediaries  and  end 
users.  The  chi-square  statistic  was  also  converted  to  a  probability 
statistic  by  SPSS  called  the  significance  level.  The  significance  levels  are 
included  with  the  chi-square  values  for  each  feature  in  the  cross -tabulations 
in  Appendix  D. 

Those  relationships  between  variables  which  are  accepted  for  this  study 
as  statistically  significant  have  a  .05  probability  of  occurring  by  chance. 
The  .05  significance  level  means  that  there  is  a  5%  possibility  that  the 
variables  are  unrelated  except  by  chance. 

When  the  value  of  .05  was  used  as  a  comparison,  three  of  the  27  features 
had  a  significance  level  less  than  .05.  These  three  features  were  then 
assumed  to  have  a  significant  level  of  difference  between  the  way 
intermediaries  and  end  users  ranked  the  features.  These  three  features  werei 


-61- 


Feature  Number  35-Choice  Between  Commands  or  Menus  (significance  level=. 01 82) 
(see  Figure  28), 

Feature  Number  38-Common  Retrieval  Language  (significance  level=.0530,  which 
is  above  but  very  close  to  .05)  (see  Figure  29),  and 

Feature  Number  40-User  Chooses  Command  Language  (significance  level=.0001) 
(see  Figure  30). 
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6  OUT  OF  8  (  75.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTEO  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .281 

RAW  CHI  SQUARE  =  10.04335  WITH  3  OEGREES  OF  FREEOOM .  SIGNIFICANCE  =  .0182 

CRAMER'S  V  ®  .56023 

CONTINGENCY  COEFFICIENT  *  .48875 

LAMBOA  (ASYMMETRIC)  *  .00000  WITH  COMENU  OEPENOENT .  =  .33333  WITH  USER  OEPENOENT . 

LAMBOA  (SYMMETRIC)  =  .12500 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  *  .16218  WITH  COMENU  OEPENOENT.  =  .28206  WITH  USER 


UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .20594 

KENOALL'S  TAU  B  =  -.07599.  SIGNIFICANCE  = 

.  3290 

KENOALL'S  TAU  C  ®  -.07422.  SIGNIFICANCE  = 

.3290 

GAMMA  =  -.13869 

SOMERS'S  0  (ASYMMETRIC)  =  -.09179  WITH  COMENU 
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SOMERS'S  0  (SYMMETRIC)  *  -.07466 

ETA  =  .08563  WITH  COMENU  OEPENOENT. 

PEARSON'S  R  =  -.08563  SIGNIFICANCE  =  .3206 

=  .56023  WITH  USER 

OEPENOENT. 
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Figure  28.  Crosstabulation  of  "Choice  Between  Commands  or  Menus 
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6  OUT  OF  8  (  75.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  =  7.68543  WITH  3  OEGREES  OF  FREEDOM.  SIGNIFICANCE  =  .0530 

CRAMER'S  V  =  .49007 

CONTINGENCY  COEFFICIENT  =  .44007 

LAMBOA  (ASYMMETRIC)  =  .05882  WITH  COMRET  DEPENDENT.  =  .22222  WITH  USER  OEPENOFNT . 
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Figure  29.  Cross tabulation  of  "Common  Retrieval  Language". 
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Figure  30*  Cross tabulation  of  "User  Chooses  Command  Language 


DISCUSSION  OF  USER-DEFINED  REQUIREMENTS 


The  results  to  questions  number  34-60,  features  which  could  be  included 
in  the  on-line  version  of  the  directory,  were  analyzed  in  order  to  identify 
the  user-defined  requirements  for  searching  the  on-line  version  of  the 
directory  on  the  Gateway. 

The  features  were  ranked  in  order  of  preference  by  the  participants  in 
the  study  as  follows.  All  responses  were  counted  and  assigned  point  values: 
The  number  one  responses  were  assigned  the  value  -2,  the  number  2  responses 
were  assigned  the  value  -1,  the  number  3  responses  were  assigned  0  points, 
the  number  4  responses  were  assigned  the  value  +1,  and  the  number  5  responses 
were  assigned  the  value  +2.  The  responses  under  each  number  in  the  scale 
were  multiplied  by  the  point  values  assigned  and  a  total  was  computed  for 
each  feature.  The  total  for  each  feature  was  the  score  assigned  to  that 
feature. 

Seventeen  "no  opinion"  responses  were  recorded  (1.96%)  and  no  feature 
received  more  than  3  “no  opinion”  responses.  It  is  assumed  that  anyone  who 
gave  a  "no  opinion”  response  had  implied,  ”1  don’t  care”.  They  could  not 
have  intended,  ”1  don’t  know,”  since  the  responses  were  gathered  over  the 
telephone,  and  any  questions  participants  had  could  be  easily  answered. 
Therefore,  in  this  study,  the  ”no  opinion”  responses  were  treated  as  3’s  and 
assigned  0  points.  They  were  not  reflected  in  the  bar  graphs,  since  they 
were  non-responses.  The  results  were  tabulated  using  the  98%  responses  which 
reflected  opinions. 

Table  9  lists  the  features  in  the  order  they  are  in  the  questionnaire. 
The  score  for  each  feature  is  in  the  right  column. 
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The  features  were  then  assembled  in  descending  order  from  highest  score 
to  lowest  score.  In  this  way,  those  features  with  the  most  responses  in  the 
upper  end  of  the  scale  were  weighted,  and  the  scores  for  those  features  were 
highest,  thus  identifying  them  as  the  most  desirable  features  to  the 
participants  of  the  study. 

Table  10  lists  the  features  in  descending  order.  The  score  for  each 
feature  is  the  number  in  parentheses  after  the  feature.  A  “Yes"  is  recorded 
in  the  column  under  an  interface  if  the  interface  possesses  the  feature  in 

that  row.  A  No  means  the  interface  does  not  possess  the  feature  in  that 
row. 

The  presence  or  absence  of  features  in  each  of  the  four  interfaces 
compared  in  Table  10  was  verified  in  conversation  by  telephone  with  Dr.  Tamas 
Doszkocs  (CITE),22  Dr.  Richard  S.  Marcus  (ASSIST  and  CONIT)23  and  Dr. 

Gabriel  Jakobson  (FRED).2^ 
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TABLE  9 


FEATURES  AND  SCORES 


FEATURE  NO. 

FEATURE 

SCORE 

34 

Minimum  Amount  of  Keyboarding 

36 

35 

Choice  Between  Commands  or  Menus 

43 

36 

Command-Driven  Only 

-12 

37 

Menu-Driven  Only 

-14 

38 

Common  Retrieval  Language 

37 

39 

Accepts  User-Defined  Commands 

0 

40 

User  Chooses  Command  Language 

33 

41 

Choose  Among  Levels  of  Expertise 

30 

42 

Accepts  Natural  English  Language 

7 

43 

Compensates  for  Spelling  Errors 

10 

44 

Questions  to  Formulate  Search  Strategy 

-4 

45 

Suggests  Related  Terms 

32 

46 

Gives  Feedback  on  Search  Strategy 

19 

47 

System  or  User  Selects  Data  Base 

23 

48 

System  Chooses  the  Data  Base 

-26 

49 

Displays  Process  it  Followed 

31 

50 

Ranks  Retrieved  Results  for  Relevancy 

16 

51 

Explains  User  Errors 

34 

52 

Includes  a  Help  Feature 

34 

53 

Stores  Search  Strategy 

42 

54 

User  Can  Define  Output  Formats 

52 

55 

Stores  User-Defined  Formats 

38 
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TABLE  9 — Continued 


FEATURE  NO. 

FEATURE 

SCORE 

56 

Has  Menu  of 

Canned  Formats 

35 

57 

Allows  User 

to  Create  Charts 

10 

58 

Allows  User 

to  Create  Graphs 

6 

59 

Allows  User 

to  Download  Information 

45 

60 

Allows  User 

to  Reformat  Information 

35 
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TABLE  10 


FEATURES  IN  ORDER  OF  PREFERENCE  BY  THE 
USERS  AND  COMPARED  TO  THE  FOUR  INTERFACES 


NO. 

FEATURE  (SCORE) 

ASSIST 

CITE 

CONIT 

FRED 

54. 

Allows  User  to  Define  Output  Formats  (52) 

Yes 

No 

Yes 

No 

59. 

Allows  User  to  Download  Information  (45) 

Yes 

No 

Yes 

No 

35. 

User  Chooses  Between  Commands  or  Menus  (43) 

Yes  -  both  at 
same  time 

No 

No 

Yes- 

Choice 

between 

natural 

language 

&  menu 

53. 

Stores  Search  Strategy  for  Later  Use  (42) 

Yes 

No 

Yes 

Yes 

55. 

Allows  Storage  of  User-Defined  Formats  (38) 

No 

No 

No 

No 

38. 

Common  Retrieval  Language  (37) 

Yes 

No 

Yes 

No 

34. 

Minimum  Amount  of  Keyboarding  (36) 

Yes 

Yes 

Yes 

Yes 

56. 

Has  a  Menu  of  Canned  Formats  (35) 

No 

No 

No 

No 

60. 

Allows  User  to  Reformat  Information  (35) 

No 

No 

No 

No 

51. 

Explains  User  Errors  (34) 

Yes 

Yes  -  hard 
to  make 

errors 

Yes 

Yes 

52. 

Includes  a  Help  Feature  (34) 

Yes 

Yes 

Yes 

Yes 

40. 

User  Chooses  Command  Language  (DROLS,  DoE ,  NASA)  (33) 

Yes 

No 

Yes 

No 

TABLE  10 — Continued 


NO. 

FEATURE  (SCORE) 

ASSIST 

CITE 

CONIT 

FRED 

45. 

Suggests  Related  Terms  (32) 

No 

Yes 

No 

Yes 

49. 

Displays  Process  it  Followed  to  Obtain  Results  (31) 

Yes 

No 

Yes 

Yes 

41. 

User  Chooses  Among  Levels  of  Searching  Expertise  (30) 

No 

No 

No 

Yes 

47. 

Choice  Between  System  or  User  Selecting  Data  Base  (23) 

No 

Yes 

No 

Yes 

46. 

Gives  Feedback  on  Search  Strategy  to  Improve  It  (19) 

Yes 

Yes-also 
accepts 
user  feed¬ 
back 

Moderately 

Yes 

50. 

Ranks  Retrieved  Results  According  to  Relevancy  (16) 

No 

Yes 

No 

No 

43. 

Compensates  for  Spelling  Errors  (10) 

No 

No-detec- 
tion,  not 
correction 

No 

Yes 

57. 

Allows  the  User  to  Create  Charts  (10) 

No 

No 

No 

No 

42. 

Accepts  Natural  English  Language  (7) 

No 

Yes 

No 

Yes 

58. 

Allows  User  to  Create  Graphs  (6) 

No 

No 

No 

No 

39. 

User-Defined  Commands  (0) 

No 

No 

No 

No 

44. 

Questions  User  to  Formulate  Search  Strategy  (-4) 

Yes 

No 

No 

No 

36. 

Command-Driven  Only  (-12) 

Yes 

No 

Yes 

No 

37. 

Menu-Driven  Only  (-14) 

Yes 

Yes 

No 

No 

48. 

System  Chooses  the  Data  Base  (-26) 

No 

No 

No 

No 

TABLE  10 — Continued 


NO. 

FEATURE  (SCORE) 

ASSIST 

CITE 

CONIT 

FRED 

45. 

Suggests  Related  Terms  (32) 

No 

Yes 

No 

Yes 

49. 

Displays  Process  it  Followed  to  Obtain  Results  (31) 

Yes 

No 

Yes 

Yes 

41. 

User  Chooses  Among  Levels  of  Searching  Expertise  (30) 

No 

No 

No 

Yes 

47. 

Choice  Between  System  or  User  Selecting  Data  Base  (23) 

No 

Yes 

No 

Yes 

46. 

Gives  Feedback  on  Search  Strategy  to  Improve  It  (19) 

Yes 

Yes-also 
accepts 
user  feed¬ 
back 

Moderately 

Yes 

50. 

Ranks  Retrieved  Results  According  to  Relevancy  (16) 

No 

Yes 

No 

No 

43. 

Compensates  for  Spelling  Errors  (10) 

No 

No-detec¬ 
tion,  not 
correction 

No 

Yes 

57. 

Allows  the  User  to  Create  Charts  (10) 

No 

No 

No 

No 

42. 

Accepts  Natural  English  Language  (7) 

No 

Yes 

No 

Yes 

58. 

Allows  User  to  Create  Graphs  (6) 

No 

No 

No 

No 

39. 

User-Defined  Commands  (0) 

No 

No 

No 

No 

44. 

Questions  User  to  Formulate  Search  Strategy  (-4) 

Yes 

No 

No 

No 

36. 

Command-Driven  Only  (-12) 

Yes 

No 

Yes 

No 

37. 

Menu-Driven  Only  (-14) 

Yes 

Yes 

No 

No 

48. 

System  Chooses  the  Data  Base  (-26) 

No 

No 

No 

No 

CONCLUSIONS 


Based  on  the  participants'  ranking  of  features  and  comparison  to  the 
four  interfaces  in  Table  10,  it  was  hoped  that  one  of  these  four  interfaces 
would  stand  out  as  being  the  most  effective  interface  for  searching  the 
on-line  version  of  the  directory  on  the  Gateway.  The  process  followed  to 
arrive  at  a  conclusion  is  described  here* 

If  a  score  of  30  is  used  as  a  cut-off  point,  a  line  can  be  drawn  on 
Table  10  under  Feature  Number  41.  This  will  distinguish  the  top  15 
user-defined  requirements  for  an  interface  for  the  directory  on  the  Gateway. 
Next,  all  the  yes  responses  are  counted  for  each  interface  compared  on  Table 
10  for  these  top  15  features;  ASSIST  has  10  yes  responses,  CITE  has  4,  C0NIT 

has  9,  and  FRED  has  8. 

CITE  is  rejected  first,  because  it  has  only  4  of  the  top  15 
user-defined  requirements  for  an  interface.  Also,  only  one  of  these  four 
requirements  can  be  considered  unique  to  CITE,  and  that  is  Feature  Number  45 

-  Suggests  Related  Terms. 

The  feature  which  is  most  unique  to  FRED  is  Feature  Number  42  —  Accepts 
Natural  English  Language.  This  feature  received  a  score  of  7,  which  places 
it  fairly  low  in  the  ranking.  Most  of  FREDfs  other  features  in  the  top  15 
are  also  shared  by  some  of  the  other  interfaces.  Therefore,  FRED  is  rejected 

at  this  point. 

C0NIT  and  ASSIST  have  many  features  which  are  ranked  high  in  the  list  of 
top  15  user-defined  requirements.  These  features  include:  Feature  Number  54 

-  Allows  the  User  to  Define  Output  Formats,  Feature  Number  59  -  Allows  the 
User  to  Download  Information,  Feature  Number  53  -  Stores  Search  Strategy  for 
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Use,  Feature  Number  38  -  Uses  a  Common  Retrieval  Language,  Feature  Number  34 
-  Minimum  Amount  of  Keyboarding,  Feature  Number  51  -  Explains  User  Errors, 
Feature  Number  52  -  Includes  a  Help  Feature,  Feature  Number  40  -  User  Chooses 
a  Command  Language,  and  Feature  Number  49  -  Displays  the  Process  It  Followed 
to  Obtain  Results. 

A  requirement  for  an  interface  ranked  highly  by  the  users  was  a  common 
retrieval  language  (Feature  Number  38)  or  a  command  language  chosen  by  the 
user  (Feature  Number  40).  One  of  CONIT's  unique  characteristics  is  a  simple, 
easy-to-leam  command  language  that  can  be  used  to  search  all  the  data  bases. 
ASSIST  is  an  enhanced  version  of  CONIT  which  allows  the  user  to  use  commands 
and  menus  at  the  same  time.  (This  requirement  -  Feature  Number  35  -  was 
ranked  3rd  by  the  participants  of  the  study  with  a  score  of  43.) 

Both  ASSIST  and  CONIT  have  been  developed  by  Dr.  Richard  S.  Marcus  at 
the  Massachusetts  Institute  of  Technology  (MIT).  CONIT  is  a  user-friendly 
on-line  search  assistance  intermediary  that  allows  for  a  minimum  amount  of 
keyboarding,  extensive  instructional  dialogue,  and  uses  a  common  command 
language  to  aid  in  searching  a  number  of  commercial  on-line  bibliographic 
data  bases.  ASSIST  is  a  new  version  of  CONIT  "designed  to  integrate  the  best 
features  of  standard  CONIT... as  well  as  some  newer  ideas,"^  which 
include  leading  a  user  through  the  entire  search  process  by  a  question  and 
answer  dialogue  with  a  menu  format,  an  on-line  tutorial  to  introduce  the  user 
to  CONIT  commands,  and  the  option  of  ASSIST  executing  CONIT  commands  for  the 
user  or  allowing  the  user  to  take  more  control  of  the  search  strategy. 
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RECOMMENDATIONS 


As  a  result  of  this  study,  I  recommend  that  the  Defense  Technical 
Information  Center  (DTIC)  contract  with  Dr.  Richard  Marcus  to  develop  a  CONIT 
or  ASSIST-like  interface  for  the  on-line  Data  Base  of  Data  Bases  on  the 
Gateway.  Such  a  custom-developed  interface  would  possess  the  features  that 
are  perceived  to  be  essential  by  the  potential  users  of  the  on-line  version 
of  the  directory  (the  top  15  requirements  of  Table  10).  Those  features  of 
CONIT  and  ASSIST  which  the  users  did  not  rank  highly  in  this  study  would  not 
need  to  be  included  in  the  new  interface. 

The  benefits  of  the  development  and  incorporation  of  this  interface  into 
the  Data  Base  of  Data  Bases  on  the  Gateway  would  be  as  follows; 

1.  DTIC  would  be  placed  in  the  forefront  of  the  developing 
technology  in  interfaces  and  human/computer  interactions. 

2.  DTIC  would  gain  experience  from  participating  in  the 
development  of  this  interface,  and  the  benefits  of  that 
experience  could  be  applied  elsewhere  in  DTIC. 

3.  A  help  feature  and  a  feature  to  explain  user  errors  on  this 
interface  would  ease  the  burden  of  providing  hotline  assistance 
to  users  of  the  Data  Base  of  Data  Bases. 

4.  The  "user-friendly"  interface  would  somewhat  mitigate  the  need 
for  DTIC  to  provide  costly  training  before  a  user  could  begin 
to  search  the  Data  Base  of  Data  Bases. 

5.  The  interface  would  reduce  the  difficulty  of  searching  by 
allowing  the  user  to  make  queries  in  a  common  language  and 
receive  results  in  the  same  language. 


-79- 


6.  More  data  bases  would  be  accessible  to  more  users  and  thus  more 
information  would  be  available. 

7.  Through  increased  use  of  the  Data  Base  of  Data  Bases,  a 
duplication  of  effort  by  users  could  be  lessened  or  avoided. 

The  custom-developed  interface  for  the  Data  Base  of  Data  Bases  on  the 

Gateway  would  contain  features  as  specified  by  the  users  in  this  study. 

However,  as  Dr.  Marcus  has  pointed  out,  "users  may  not  be  able  to  predict 

usage  modes  especially  where  new  designs  and  functionality  are  desirable 
■»26 

and  likely.”  The  potential  users  in  this  study  have  indicated  which 
features  seem  desirable  to  them,  but  they  may  have  ranked  a  feature 
conservatively  if  it  was  one  they  had  never  heard  of,  or  could  not  imagine, 
or  did  not  believe  that  present  technology  would  support.  A  comment  often 
made  by  participants  when  a  particularly  sophisticated  feature  was  mentioned 
was,  That  would  be  great,  if  it  worked.”  Many  of  the  participants  could  not 
believe  that  all  the  features  mentioned  in  questions  number  34-60  of  the 
questionnaire  were  possible  in  an  interface  or  the  host  computer  within  which 
the  interface  would  reside.  Therefore,  I  recommend  that  the  final  decision 
on  the  type  of  interface  that  is  used  for  the  on-line  version  of  the 
directory  on  the  Gateway  be  made  by  those  closely  associated  with  the 
directory  and  the  Gateway.  But  the  wishes  of  the  potential  users,  as 
represented  in  this  study,  should  weigh  heavily  in  that  decision-making 
process. 

As  a  methodology,  this  study  was  exploratory  in  nature.  Due  to  the 
method  employed-mailing  out  a  standardized  questionnaire  and  collecting 
responses  over  the  telephone-a  high  number  of  responses  were  gathered  in  a 
limited  period  of  time,  and  participants  who  had  questions  were  more 
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likely  to  ask  them  than  they  would  have  been  if  they  were  not  contacted 
directly.  Therefore,  the  results  of  this  study  can  be  considered  credible. 
I  recommend  this  methodology  for  similar  studies,  however,  a  broader  sample 
group,  especially  of  end  users,  should  be  used  when  possible. 


-81- 


Appendix  A  -  Questionnaire 


1.  Name  of  Respondent: 


2.  Name  of  Organization: 


Hi!  My  name  is  Georgene  Chastain,  and  Ifm  calling  from  the  Defense 
Technical  Information  Center.  I  am  working  on  a  project  to  identify 
user  requirements  for  searching  a  Directory  of  DoD-Sponsored  R&D  Data 
Bases  on  the  Defense  Gateway  Computer  System.  You  were  suggested  as  a 
person  likely  to  have  an  interest  in  the  directory  and  the  gateway. 

3.  Have  you  had  an  opportunity  to  look  at  the  directory? 

□  NO  |  "  |  YES  (Go  to  6) 

4.  Would  you  be  willing  to  answer  some  questions  about  it  at  a  future 
date? 


NO  (Go  to  CLOSING)  □  YES 

5.  The  questions  will  take  about  15  minutes  of  your  time.  When  would 

be  the  best  time  to  call  you?  _ 

6.  The  questions  will  take  about  15  minutes  of  your  time.  Is  now  a 
good  time? 

□  NO  (Go  to  7)  YES  (Go  to  8) 

7.  When  would  be  a  good  tine  to  call  you?  _ 
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8.  Approximate  number  of  people  in  organization: 


9.  Are  you  a  member  or  supervisor  of  the  library  staff? 

□  NO  (Go  to  13)  □  YES 

10.  How  many  individuals  in  the  library  do  on-line  searching  of  data 
bases? 


11.  Would  you  please  name  the  data  bases  which  you  and/or  your  staff 
search. 


12.  I  am  going  to  read  a  short  list  of  hardware.  Please  indicate 
whether  or  not  you  have  any  of  each  type  available  to  you  and/or  your 
staff. 

Dedicated  terminal  (e.g.  Uniscope-DROLS,  Beehive-OCLC) 

□  NO  □  YES  How  Many? _ 

Dial-up  terminal  (e.g.  TI  Silent  700,  HP,  etc.) 

□  NO  □  YES  How  Many? _ 
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Communi eating  microcomputers  (NOTE:  a  microcomputer  with  a  smart 
modem  or  an  acoustic  coupler  and  communications  software.) 


o 

5S 

□ 

1  1  yes 

How  Many? 

Dedicated  microcomputers 

o 

□ 

|  f  YES 

How  Many? 

(Go  to  18) 

13.  I  am  going  to  read  you 

a  list  of  terms  which  can  be  used  to 

describe  professional  functions.  Please 

indicate  whether  or  not  i 

term  describes  your  present 

work. 

N 

Y 

Planner 

□ 

□ 

Marketer 

□ 

□ 

Scientist  or  Engineer 

□ 

□ 

Other 

14.  Do  you  do  on-line  searching  of  data  bases? 

□  NO  (Go  to  17)  □  YES 

15.  Would  you  please  name  the  data  bases  which  you  search? 
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16.  I  am  going  to  read  a  short  list  of  hardware.  Please  indicate 
whether  or  not  you  have  any  of  each  type  available  to  you  and/or  your 
staff. 

Dedicated  terminal  (e.g.  Uniscope-DROLS,  Beehive-OCLC) 

□  NO  □  YES  How  Many? 

Dial-up  terminal  (e.g.  TI  Silent  700,  HP,  etc.) 

□  NO  □  YES  How  Many? _ 

Communicating  microcomputers  (NOTE:  a  microcomputer  with  a  smart 
modem  or  an  acoustic  coupler  and  communications  software.) 

NO  □  YES  How  Many? _ 

Dedicated  microcomputers 

□  NO  □  YES  How  Many? _ 

(Go  to  18) 

17.  Who  does  on-line  searching  of  data  bases  for  you? 
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18.  The  Directory  of  DoD-Sponsored  R&D  Data  Bases  is  a  listing  of 


DoD 1 s  R&D  data  bases.  Each  entry  contains  information  such  as  the 
data  base  name,  dates  of  coverage,  points  of  contact, 
hardware/software  configuration,  and  a  description  of  the  data  base. 
Agency,  data  base,  and  subject  indices  are  provided.  The  subject 
coverage  includes  meteorology,  weapon  systems,  hazardous  materials, 
medicine,  oceanography,  antennas,  survivability,  reliability,  and 
chemistry.  Do  you  presently  use  the  Directory  of  DoD-Sponsored  R&D 
Data  Bases? 

|  |  NO  (Go  to  20)  |  |  YES 

19.  I  am  going  to  read  you  a  list  of  the  indices  which  are  in  the 


directory.  Please  indicate 

use  each  index. 

whether  or 

N 

not  you  have  had  occasion  to 

Y 

Data  Base  Index 

□ 

□ 

Organization  Index 

□ 

□ 

Subject  Index 

□ 

□ 

20.  Do  you  expect  to  be  using  the  directory  in  the  future? 

j  |  NO  (Go  to  22)  □  YES 

21.  If  the  directory  were  available  on-line,  would  you  still  need  the 
hard  copy  directory? 

I  |  NO  Q  YES 
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22.  If  the  directory  were  available  on-line  and  accessible  through 
dial-up  terminals,  would  your  organization  use  it? 

□  NO  □  YES 

The  Defense  Gateway  Computer  System  is  being  developed  to  make  it 
easier  to  access,  reformat,  and  analyze  information  from  data  bases 
of  interest  to  the  DoD  community.  At  the  present  time,  a  capability 
has  been  developed  to  automatically  access,  reformat,  and  analyze 
information  in  the  Defense  RDT&E  On-Line  System  (DROLS),  NASA/RECON, 
and  DOE /RECON.  The  Directory  of  DoD-Sponsored  R&D  Data  Bases  will 
eventually  be  made  available  on-line  on  the  gateway.  This  on-line 
version  will  be  called  the  data  base  of  data  bases.  Some  of  the  data 
bases  described  in  the  directory  will  be  accessible  through  the 
gateway. 

23.  When  the  directory  is  available  on-line  through  the  gateway,  will 
you  be  the  one  searching  it? 

□  NO  □  YES  (Go  to  25) 

24.  (a)  Would  you  want  someone  to  search  it  for  you? 

□  NO  (Go  to  CLOSING)  □  YES 

(b)  Please  give  me  the  name  of  the  person  who  will  be  searching 

it. 
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25.  (If  the  respondent  is  a  member  or  supervisor  of  the  library  staff 
go  to  26,  else  go  to  27.)  [see  9] 


26.  If  the  on-line  directory  were  easy  to  use,  do  you  think  your 
library  users  and/or  researchers  would  use  it? 


□  N0  □  YES 

27.  I  am  going  to  read  a  list  of  the  fields  in  the  directory.  After  I 
have  read  each  field  name,  please  indicate  whether  or  not  you  would  be 


likely  to  search  that  field. 


Data  Base  Name: 

□ 

□ 

Acronym: 

□ 

□ 

Update  frequency: 

□ 

□ 

Beginning  date: 

□ 

□ 

Ending  date: 

□ 

□ 

Size : 

□ 

□ 

Data  Base  Producer  Name: 

□ 

□ 

Data  Base  Producer  Address: 

□ 

□ 

Data  Base  Producer  Contact: 

□ 

□ 
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N 

Y 

Data  Base  Distributor  Name: 

□ 

□ 

Data  Base  Distributor  Address: 

□ 

□ 

Data  Base  Distributor  Contact: 

□ 

□ 

Data  Base  Generator  Name: 

□ 

□ 

Data  Base  Generator  Address: 

□ 

□ 

Data  Base  Generator  Contact: 

□ 

□ 

Availability  of  the  data  base: 

□ 

□ 

Descriptors : 

□ 

□ 

Data  Base  Type: 

□ 

□ 

Code  Character  Set: 

□ 

□ 

Density: 

□ 

□ 

Number  of  Tracks: 

□ 

□ 

Labeled: 

□ 

□ 

Programming  Language: 

□ 

□ 

Computer: 

□ 

□ 

Storage  Media: 

□ 

□ 

Input  Media: 

□ 

□ 
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N 

Y 

Output  Media: 

□ 

□ 

Documentation : 

□ 

□ 

Classification  Restrictions: 

□ 

□ 

Abstract: 

□ 

□ 

28.  Are  there  any  additional  fields  which  you  would  like  to  see 
included  in  the  on-line  version? 

□  NO  (Go  to  30)  □  YES 

29.  Please  describe  those  additional  fields. 


30.  (a)  Are  there  any  fields  in  the  directory  which  you  would  not  want 
to  display? 


□  NO  (Go  to  31)  □  YES 

(b)  Which  ones? 

Data  Base  Name: 

Acronym: 

Update  frequency: 


□ 

□ 

□ 
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Beginning  date: 


Ending  dates 

Size : 

Data  Base  Producer  Name: 

Data  Base  Producer  Address: 
Data  Base  Producer  Contact: 
Data  Base  Distributor  Name: 
Data  Base  Distributor  Address: 
Data  Base  Distributor  Contact: 
Data  Base  Generator  Name: 

Data  Base  Generator  Address: 
Data  Base  Generator  Contact: 
Availability  of  the  data  base: 
Descriptors: 

Data  Base  Type: 

Code  Character  Set: 

Density: 


□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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Number  of  Tracks: 


Number  of  Tracks: 

□ 

Labeled: 

□ 

Programming  Language: 

□ 

*  Computer: 

□ 

Storage  Media: 

□ 

Input  Media: 

□ 

Output  Media: 

□ 

Documentation : 

□ 

Classification  Restrictions: 

□ 

Abstract: 

□ 

31.  Are  there  any  additional  fields  which  you  would  like  to  see 
included  in  the  on-line  version? 

1  I  NO  (Go  to  33)  □  YES 

32.  Please  describe  those  additional  fields. 
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33.  I  am  going  to  read  a  list  of  features  which  could  be  included  in 
the  on-line  version  of  the  directory.  I  would  like  you  to  rank  each 
feature  on  a  scale  of  one  to  five,  one  meaning  that  the  feature  would 
not  be  useful  for  you  and  five  meaning  that  the  feature  would  be 
essential  for  you. 


34.  a  system  which  requires  a  minimum  amount  of  keyboarding. 


1 

2 

3 

4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

35.  a  system 

that  allows 

the  user  to 

1  use  either  commands  or  menus, 

1 

2 

3 

4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

36.  a  system 

that  is  command-driven 

only. 

1 

2 

3 

4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

37.  a  system 

that  is  menu 

-driven  only. 

1 

2 

3 

4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

38.  a  system  with  a  common  retrieval 

language 

using 

simple 

commands • 

1 

2 

3 

4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

-94- 


39.  a  system  that  accepts  user-defined  commands. 

^  ^  3  4  5  No  Opinion 

□  □□□□□ 

40.  a  system  that  allows  the  user  to  choose  among  command  languages. 
Thus  the  user  could  use  the  command  language  of  D0E/REC0N,  NASA/RECON, 
DIALOG,  DROLS,  etc.  to  search  the  directory. 


1 

2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 

□ 

□ 

41.  a  system 

that  allows  the  user  to  choose  among  levels  of  searching 

expertise,  for  example, 

beginner,  intermediate,  expert 

• 

i 

2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 

□ 

□ 

42.  a  system 

that  accepts  unstructured  natural  English 

language 

queries  and  gives  natural  English  language 

replies. 

1 

2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 

□ 

□ 

43.  a  system 

that  compensates  for  spelling 

errors. 

1 

2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 

□ 

□ 

44.  a  system  which  poses 

questions  to  the  user  and  uses 

i  the  responses 

to  formulate 

the  search 

strategy. 

1 

2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 

□ 

□ 
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45.  a  system  that  can  suggest  related  terms  for  your  search 
strategy. 


1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

46.  a  system  that  gives 

feedback  on  the  search  strategy,  suggesting 

ways  to  improve  it. 

1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

47.  a  system  which  allows  the  user  to  choose 

between 

the  system 

selecting  the  data  base 

or  the  user  selecting 

the  data  base  to 

search. 

1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

48.  a  system  that  automatically  decides  which 

data  base  to  search. 

1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

49.  a  system  that  displays  the  process  it  followed  in 

order  to  obtain 

the  search  results. 

1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□ 

□ 

□ 

□ 

50.  a  system  which  ranks 

retrieved  items  according  to 

their 

relevancy • 

1  2 

3  4 

5 

No  Opinion 

□ 

□ 

□  □ 
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□ 

□ 

system  which  explains  user  errors. 


12  3  4 

□  □  □  □ 


5  No  Opinion 

□  □ 


system  which  includes  a  help  feature. 


12  3  4 

□  □  □  □ 


□ 


No  Opinion 

□ 


system  which  stores 


the  search  strategy  for  later  use. 


□  □ 


3  4 


5  No  Opinion 


□  □ 


□  □ 


system  which  allows  the  user  to  define  output  formats. 


□  □ 


3  4  5  No  Opinion 

□  □  □  □ 


system  which  allows  the  storage  of  user- defined  formats. 


1  2  3  4  5 

□  □  □  □  □ 

« 

system  which  has  a  menu  of  canned  formats. 

1  2  3  4  5 

□  □  □  □  □ 

system  which  allows  the  user  to  create  charts. 
1  2  3  4  5 


No  Opinion 

□ 


No  Opinion 

□ 


No  Opinion 


58.  a  system  which  allows  the  user  to  create  graphs. 

^  ^  3  4  5  No  Opinion 

□  □□□□□ 

59.  a  system  which  allows  the  user  to  download  information. 

^  ^  3  4  5  No  Opinion 

□  □□□□□ 

60.  a  system  which  allows  the  user  to  reformat  the  information. 

^  3  4  5  No  Opinion 

□  □□□□□ 

61.  Are  there  any  additional  requirements  which  you  would  like  to  see 
included  in  the  on-line  version? 

I  I  NO  (Go  to  CLOSING)  YES 

62.  Please  describe  those  additional  requirements. 


That  s  all  the  questions  X  have  to  ask  you.  You  have  been  very 
helpful.  Thank  you  very  much.  Good-bye! 


-98- 


Appendix  B 

List  of  Users  Questioned 


Mr.  H.  Eugene  Thompson 
Office  of  the  Director 
Defense  Test  and  Evaluation 
The  Pentagon,  Room  3D973 
Washington,  DC  20301 


Mr.  H.F.  Hege 

Chemical  Propulsion  Information  Agency 
Johns  Hopkins  University 
Applied  Physics  Laboratory 
Johns  Hopkins  Road 
Laurel,  MD  20707 


Marilyn  Harned  and  Pat  Prentice 
Naval  Air  Systems  Command  Library 
Attn:  AIR-7226 
Washington,  DC  20361 


Betsy  L.  Fox 
Defense  Nuclear  Agency 
Attn:  STTI 

Washington,  DC  20305 


Mary  B.  Vick 
USASCAF 

The  Pentagon  Library 
The  Pentagon,  Room  1A518 
Washington,  DC  20310 


John  Petrone 

Army  Industrial  Base  Engineering  Activity 
Rock  Island  Arsenal 
Rock  Island,  IL  61299 


Andrej  Bevec 

Harry  Diamond  Laboratories 
Attn:  Branch  21100 
2800  Powder  Mill  Road 
Adelphi,  MD  20783 
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Frank  Hamden 

DLSIE-Defense  Logistics  Studies  Information  Exchange 

Army  Logistics  Management  Center 

DRXMC-D 

Fort  Lee,  VA  23801 


Linda  Evans 

Rome  Air  Development  Center 
Technical  Services 
Griffiss  Air  Force  Base 
Rome,  NY  13441 


Sarah  Happel 

Naval  Surface  Weapons  Center 
Technical  Library 
White  Oak 

Silver  Spring,  MD  20910 


Sandy  Rose 

Naval  Surface  Weapons  Center 
Technical  Library 
Dahlgren  Laboratory 
Dahlgren,  VA  22448 


Mary  R.  Weston 
U.S.  Air  Force 

Technical  Library  AFATL/DLODL 
Eglin  Air  Force  Base 
Eglin,  FL  32542 


Burt  Newlin 

Defense  Material  Specifications  & 
Standards  Office 
5203  Leesburg  Pike,  Suite  1403 
Falls  Church,  VA  22041-3466 


Frank  Jones 
U.S,  Air  Force 

Wright  Aeronautical  Laboratory 
Wright-Patterson  Air  Force  Base 
Dayton,  OH  45433 
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Linda  Cheung 

U.S.  Army  Foreign  Science  & 
Technology  Center 
Information  Services  Division 
AMSXT-IS3 

220  Seventh  Street,  NE 
Charlottesville,  VA  22901 


Charles  Maiorana 
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Appendix  C  -  Histograms 
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******************)(i)^)|r)(ijf)^)^  +  ]f  +  +  +^ 


I  ESSENTIAL 
I 

I . I . I 

0  4  8 

FREQUENCY 


.1 

12 


12) 


13) 

.  I.  . 

16 


.1 

20 


MEAN 

4 

.094 

MODE 

5 

.000 

KURTOSIS 

1  , 

.740 

MINIMUM 

1  . 

.000 

VALID  CASES 

32 

STD  ERR  .  176 
STO  OEV  .995 
SKEWNESS  -1.244 
MAXIMUM  5.000 

MISSING  CASES  0 


MEOIAN 

4.250 

VARIANCE 

.991 

RANGE 

4.000 

CHARTS 

COOE 

1 

2 

3 

4 

5 


USER  CREATE  CHARTS 
I 

******  (  2) 

I  NOT  USEFUL 
I 

****************  (  6) 

I  SOMEWHAT  USEFUL 
I 

*****************************  ^ 
I  INDIFFERENT 
I 

****************  (  6) 

I  VERY  USEFUL 
I 

*******************  ^  7) 

I  ESSENTIAL 
I 

I . I . I . I 

0  4  8  12 

FREQUENCY 


11) 


.  I 
16 


.  I 
20 


MEAN 

3 

.313 

MODE 

3 

.000 

KURTOSIS 

.825 

MINIMUM 

1  , 

.000 

VALIO  CASES 

32 

STO  ERR 

.213 

STD  OEV 

1.203 

SKEWNESS 

-.060 

MAXIMUM 

5.000 

MISSING  CASES 

0 

MEDIAN  3.227 
VARIANCE  1.448 
RANGE  4.000 


-127 


GRAPHS 

CDDE 

1 

2 

3 

4 

5 


USER  CREATE  GRAPHS 

I 

******(  2 ) 

I  NDT  USEFUL 
I 

*********************  (  g) 

I  SOMEWHAT  USEFUL 
I 

*****************************  ^ 
I  INDIFFERENT 
I 

***********  (  4  ) 

I  VERY  USEFUL 
I 

*******************  (  7) 

I  ESSENTIAL 
I 

I . I . I . I 

0  4  8  12 

FREQUENCY 


11) 


.  I 
16 


.  I 
20 


MEAN 

3. 

.  188 

MODE 

3. 

•  OOO 

KURTDSIS 

.933 

MINIMUM 

1  . 

OOO 

VALID  CASES 

32 

STD  ERR  .217 
STD  DEV  1.230 
SKEWNESS  .174 
MAXIMUM  '  5.000 

MISSING  CASES  O 


MEDIAN  3.045 
VARIANCE  1.512 
RANGE  4.000 


DOWNLD 

CDDE 

2 

3 

4 

5 


DOWNLOAD  INFDRMATI DN 
I 

****  (  1) 

I  SOMEWHAT  USEFUL 
I 

***********  (  4) 

I  INDIFFERENT 
I 

*********************  ^ 
I  VERY  USEFUL 
I 


I  ESSENTIAL 
I 

1 . I . I . I . I . 

0  4  8  12  16 

FREQUENCY 


**  ( 


.1 

20 


19) 


MEAN 

4.406 

MDDE 

5.000 

KURTDSIS 

.814 

MINIMUM 

2.000 

VALID  CASES 

32 

STD  ERR  . 148 
STD  DEV  .837 
SKEWNESS  -1.268 
MAXIMUM  5.000 

MISSING  CASES  0 


MEDIAN  4.658 
VARIANCE  .701 
RANGE  3.000 


-129- 


REFORM 

CODE 


REFORMAT 


INFORMATION 


****  (  1) 

I  NOT  USEFUL 
I 

*********  (  3) 

I  SOMEWHAT  USEFUL 
I 

****** . .  4) 

I  INDIFFERENT 
I 

•  *•*••••*******•***•41  ( 

I  VERY  USEFUL 
I 


8) 


I  ESSENTIAL 
I 

I . I.  . 

O  4 

FREQUENCY 


16) 


.1. 

12 


.1. 

16 


.  .1 

20 


MEAN 

MODE 

KURTOSIS 

MINIMUM 

4.094 
5.000 
.448 
t  .OOO 

STD  ERR 

STD  DEV 

SKEWNESS 

MAXIMUM 

.203 

1.  146 
-1.154 
5.000 

MEDIAN 

VARIANCE 

RANGE 

4.500 

1.314 

4.000 

VALID  CASES 

32 

MISSING  CASES 

0 

131 


I 

U) 

VjO 

I 


******************  CRDSST  ABULAT  IDN  OF  **** 
KEY  MIN  AMT  DF  KEYBDAROING  BY  USER  USER  TYPE 

*********************  +  3tt  +  +  ^  +  +  +  3tt  +  3(t  +  +  +  Jtt3tt  +  +  +  +  :(t 


USER 

CDUNT  I 

RDW  PCT  I INTERMED  END-USER  RDW 


CDL 

PCT 

1 1 ARY 

TOTAL 

TOT 

PCT 

I 

1 

I 

2 

I 

!Y  - 

3 

I 

6 

I 

2 

I 

8 

INDIFFERENT 

I 

75.0 

I 

25.0 

I 

25.0 

I 

26.  1 

I 

22.2 

I 

I 

18.8 

I 

6.3 

I 

I- 

-I- 

-I 

4 

I 

9 

I 

3 

I 

12 

VERY  USEFUL 

I 

75.0 

I 

25.0 

I 

37.5 

I 

39. 1 

I 

33.3 

I 

I 

28.  1 

I 

9.4 

I 

I- 

-I- 

-I 

5 

I 

8 

I 

4 

I 

12 

ESSENTIAL 

I 

66.7 

I 

33.3 

I 

37.5 

I 

34.8 

I 

44.4 

I 

I 

25.0 

I 

12.5 

I 

- 

I- 

-I- 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28.  1 

100.0 

****** 


*  *  *  PAGE 


3  DUT  DF  6  (  50.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  2.250 


RAW  CHI  SQUARE  =  .25765  WITH 

CRAMER ' S  V  *  .08973 

CONTINGENCY  COEFFICIENT  =  .08937 

LAMBDA  (ASYMMETRIC)  =  .05000  WITH  KEY 

LAMBDA  (SYMMETRIC)  =  .03448 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  * 
UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  = 
KENDALL'S  TAU  B  =  .07584.  SIGNIFICANCE  = 

KENDALL #  S  TAU  C  =  .07813.  SIGNIFICANCE  = 

GAMMA  =  . 14706 

SOMERS'S  D  (ASYMMETRIC)  =  .09662  WITH  KEY 

SDMERS'S  D  (SYMMETRIC)  =  .07366 

ETA  =  .07791  WITH  KEY  DEPENDENT. 

PEARSDN'S  R  =  .07791  SIGNIFICANCE  =  .3358 


2  DEGREES  DF  FREEDOM.  SIGNIFICANCE 


.8791 


DEPENDENT. 

.00368  WITH  KEY 
.00475 

.3275 
.3275 

DEPENDENT. 


.00000  WITH  USER 


DEPENDENT. 


DEPENDENT. 


.00670  WITH  USER 


.08973  WITH  USER 


.05952  WITH  USER 


DEPENDENT. 


DEPENDENT. 


*  *  * 

1  OF  1 


DEPENDENT. 


Appendix  D  -  Cross tabulations 


***************** 


*************** 


*  * 


crosstabulation  of 

CDMENU  COMMANOS  OR  MENUS  BY  USER  USER  TYPE 

*************************************************  PAGE 


VjJ 

vn 

l 


COUNT 
ROW  PCT 
COL  PCT 
TOT  PCT 


COMENU 


NOT  USEFUL 


INDIFFERENT 


VERY  USEFUL 


ESSENTIAL 


USER 

I 

I INTERMEO  ENO-USER 

I I  ARY 

I  II  2 

I - 1 - 


1 


COLUMN 

TOTAL 


1 

100.0 
4.3 
3.  1 


O 

.0 

.0 

.0 


10 
90.9 
43.5 
31 .3 


O 

.0 

.0 

.0 


3 

100.0 

33.3 

9.4 


1 

9.  1 
11.1 
3.  1 


ROW 

TOTAL 


1 

3.  1 


3 

9.4 


1  1 
34.4 


12 

I 

5 

I  17 

70.6 

I 

29.4 

I  53.1 

52.2 

I 

55.6 

I 

37.5 

I 

15.6 

I 

_  T 

23 

71.9 


9 

28. 1 


32 

100.0 


6  OUT  OF  8  (  75.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTEO  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTEO  CELL  FREQUENCY  *  .281 

RAW  CHI  SQUARE  =  10.04335  WITH  3  DEGREES  OF  FREEDOM.  SIGNIFICANCE  *  .0182 

CRAMER ' S  V  =  .56023 

CONTINGENCY  COEFFICIENT  =  .48875 

LAMBOA  (ASYMMETRIC)  =  .00000  WITH  COMENU  DEPENDENT.  =  .33333  WITH  USER  DEPENDENT. 

LAMBOA  (SYMMETRIC)  =  .12500 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  *  .16218  WITH  COMENU  DEPENDENT.  =  .28206  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  *  .20594 

.3290 
.3290 


.07599. 

.07422. 


SIGNIFICANCE  = 
SIGNIFICANCE  = 


KENOALL'S  TAU  B  = 

KENOALL'S  TAU  C  = 

GAMMA  =  -.13869 

SDMERS ' S  D  (ASYMMETRIC)  =  -.09179  WITH  COMENU 

SOMERS'S  0  (SYMMETRIC)  =  -.07466 

ETA  =  .08563  WITH  COMENU  DEPENOENT. 

PEARSON'S  R  =  -.08563  SIGNIFICANCE  =  .3206 


OEPENOENT . 


.56023  WITH  USER 


-.06291  WITH  USER 


OEPENOENT. 


OEPENOENT . 


1  OF  1 


OEPENOENT. 


*  *  +  * 


*  * 


******* 


************„*  CROSSTABULATION  OF  *********** 
COM  COMMANO  ONLY  gy  USER  USER  TYPE 

***************************  *  *  *******************  * 


PAGE 


COUNT 
ROW  PCT 
COL  PCT 
TOT  PCT 


COM 


1 


NOT  USEFUL 


USER 

I 

I INTERMEO  ENO-USER  ROW 
I I ARY  TOTAL 

I  II  2  1 

l - I - i 

I  4  1  2  1  6 

I  66.7  I  33.3  I  18.8 

I  17.4  I  22.2  I 

I  12.5  I  6.3  I 


-I- 

2  I 

7 

I 

4 

I 

1  1 

USEFUL  I 

63.6 

I 

36.4 

I 

34.4 

I 

30.4 

I 

44.4 

I 

I 

21.9 

I 

12.5 

I 

-I- 

INDIFFERENT 


VERY  USEFUL 


ESSENTIAL 


8 

88.9 

34.8 

25.0 


1 

100.0 
4.3 
3.  1 


1  1 


11.1 

3.  1 


0 

.0 

.0 

.0 


3 

I 

2 

60.0 

I 

40.0 

13.0 

I 

22.2 

9.4 

I 

6.3 

COLUMN 

TOTAL 


9 

28.  1 


1 

3.  1 


5 

15.6 


23 
71 .9 


9 

28.  1 


32 

100.0 


8  OUT  OF  10  (  80.0%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5  O 
MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .281  utbb  mAN  b.U. 

RAW  CHI  SQUARE  =  2.47874  WITH  4  DEGREES  OF  FREEDOM. 

CRAMER ' S  V  =  .27832 

CONTINGENCY  COEFFICIENT  =  .26813 

LAMBDA  (ASYMMETRIC)  =  .04762  WITH  COM  DEPENDENT 

LAMBOA  (SYMMETRIC)  =  .03333 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .03217  WITH  COM 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .04551 

KENOALL  * S  TAU  B  =  -.08201.  SIGNIFICANCE  = 

KENDALL  * S  TAU  C  =  -.08984.  SIGNIFICANCE  = 

GAMMA  =  -.14650 

SOMERS ' S  0  (ASYMMETRIC)  =  -.11111  WITH  COM 

SOMERS  * S  0  (SYMMETRIC)  =  -.07836 

ETA  =  .03424  WITH  COM  OEPENDENT . 

PEARSON  * S  R  =  -.03424  SIGNIFICANCE  =  .4262 


SIGNIFICANCE  =  .6484 

=  .OOOOO  WITH  USER  OEPENOENT . 

OEPENOENT .  =  .07775  WITH  USER 


.3084 

.3084 

OEPENOENT . 


.27832  WITH  USER 


-.06053  WITH  USER 
OEPENOENT. 


OEPENOENT. 


1  OF  1 


DEPENOENT. 


UJ 

VO 

I 


****************** 

MENU  MENU  ONLY 

******************** 


CRDSSTABULATION  OF  ************** 

BY  USER  USER  TYPE 

***************************)t:5k  PAGE 


COUNT 
ROW  PCT 
COL  PCT 
TDT  PCT 


MENU 


NOT  USEFUL 


1 


SDMEWHAT  USEFUL 


INDIFFERENT 


USER 

I 

I INTERMEO  ENO-USER  ROW 
I I ARY  TDTAL 

I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 
I 

I  5  1  O 

I  100.0  I  .0 

I  21.7  I  .0 

I  15.6  I  .0 


1 

I 

2 

I 

6 

I 

2 

I 

8 

75.0 

I 

25.0 

I 

25.0 

26.  1 

I 

22.2 

I 

18.8 

I 

6.3 

I 

7 

I 

3 

I 

10 

70.0 

I 

30.0 

I 

31.3 

30.4 

I 

33.3 

I 

21.9 

i 

HH  HH 

1 

9.4 

I 

VERY  USEFUL 


ESSENTIAL 


-I- 

I 

4 

I 

2 

I 

66.7 

I 

33.3 

I 

17.4 

I 

22.2 

I 

-I- 

12.5 

I 

6.3 

1 

33.3 
4.3 
3.  1 


COLUMN 

TOTAL 


2 

66.7 

22.2 

6.3 


5 

15.6 


6 

18.8 


3 

9.4 


23 

71.9 


9 

28 . 1 


32 

100.0 


MINIMuS^XPECTEo'cEiL^RESliENCY7^  ^844  CELLS  EXPECTE°  CELL  FREQUENCY  LESS  THAN  5.0. 

^d«ucoLSwUARE  =  4.29758  WITH  4  OEGREES  OF  FREEOOM.  SIGNIFICANCE  =  .3672 

CRAMER'S  V  =  .36647 

CONTINGENCY  COEFFICIENT  =  .34409 

LAMBOA  (ASYMMETRIC)  =  .OOOOO  WITH  MENU  DEPENOENT.  -  mu  WITH  USER  OEPFNDPNT 

LAMBDA  (SYMMETRIC)  =  .03226  -uni  UbEK  DEPENDENT. 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .05445  WITH  MENU  OEPENOENT  =  14076  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .07853  .14076  WITH  USER 

.  2450 
.2450 


.11191. 
. 12500. 


SIGNIFICANCE 

SIGNIFICANCE 


KENOALL ' S  TAU  B  = 

KENOALL'S  TAU  C 
GAMMA  =  .19512 

SDMERS ' S  0  (ASYMMETRIC)  =  .15459  WITH  MENU 

SDMERS ' S  0  (SYMMETRIC)  =  .10631 

ETA  =  .15735  WITH  MENU  OEPENOENT. 

PE ARSDN ' S  R  -  .15735  SIGNIFICANCE  =  .1949 


OEPENOENT. 

-  .36647  WITH  USER 


.08101  WITH  USER 
OEPENOENT. 


DEPENDENT. 


*  *  * 

1  OF  1 


DEPENDENT. 


****************** 


****************** 


CROSSTABULATION  OF 
COMRET  CDMMON  RETRIEVAL  LANG  BY  USER  USER  TYPE 

*************************************************  PAGE 

USER 

COUNT  I 

ROW  PCT  I INTERMED  ENO-USER  ROW 


COL 

TOT 

rnMDCT  _ _ _ . 

PCT 

PCT 

1 1  ARY 

I  1 

I 

-  T  - 

2 

I 

TOTAL 

tUMKt 1  - 

i 

1 

-  1 

1 

I 

0 

I 

2 

I 

2 

NOT  USEFUL 

I 

.0 

I 

100.0 

I 

6.3 

I 

.0 

I 

22.2 

I 

I 

.0 

I 

6.3 

I 

I- 

-I- 

-I 

3 

I 

2 

I 

2 

I 

4 

INDIFFERENT 

I 

50.0 

I 

50.0 

I 

12.5 

I 

8.7 

I 

22.2 

I 

I 

6.3 

I 

6.3 

I 

I- 

-I- 

-I 

4 

I 

8 

I 

3 

I 

1  1 

VERY  USEFUL 

I 

72.7 

I 

27.3 

I 

34.4 

I 

34.8 

I 

33.3 

I 

I 

25.0 

I 

9.4 

I 

I- 

-I- 

-I 

5 

I 

13 

I 

2 

I 

15 

ESSENTIAL 

I 

86.7 

I 

13.3 

I 

46.9 

I 

56.5 

I 

22.2 

I 

I 

40.6 

I 

6.3 

I 

I- 

-I- 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28. 1 

100.0 

VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 
.  563 

3  OEGREES  OF  FREEODM .  SIGNIFICANCE  =  .0530 


6  DUT  OF  8  (  75.0%)  OF  THE 

MINIMUM  EXPECTEO  CELL  FREQUENCY  = 

RAW  CHI  SQUARE  =  7.68543  WITH 

CRAMER  * S  V  =  .49007 

CONTINGENCY  COEFFICIENT  =  .44007 

LAMBOA  (ASYMMETRIC)  =  .05882  WITH  CDMRET 

LAMBOA  (SYMMETRIC)  =  .11538 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  »  .10559  WITH  CDMRET  OEPENOENT . 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .13946 

KENOALL ' S  TAU  B  =  *.38702.  SIGNIFICANCE  = 

KENOALL  ' S  TAU  C  =  -.39453.  SIGNIFICANCE  = 

GAMMA  =  -.66013 

SOMERS' S  0  (ASYMMETRIC)  =  -.48792  WITH  COMRET 

SDMERS  ' S  0  (SYMMETRIC)  =  -.37687 

ETA  =  .48386  WITH  COMRET  OEPENDENT . 

PEARSON'S  R  =  -.48386  SIGNIFICANCE  =  .0025 


OEPENOENT. 


.01  10 
.01  10 


22222  WITH  USER 


DEPENDENT 


.20534  WITH  USER 


DEPENOENT. 


.49007  WITH  USER 


-.30699  WITH  USER 


DEPENDENT. 


OEPENDENT. 


1  OF  1 


DEPENOENT. 


******************  CROSSTABULATIDN  DF  **************< 
USRCDM  USER-DEFINED  CDMMANOS  BY  USER  USER  TYPE 

**************************  +  +  *  +  +  **  +  *  +  ***  +  *  +  +  +  +  +  +  jtt  +  PAGE 


COUNT 
RDW  PCT 
CCL  PCT 
TDT  PCT 


USRCDM 


1 


NOT  USEFUL 


VERY  USEFUL 


ESSENTIAL 


USER 

I 

I I NTERMED  END-USER 

I I  ARY 

I  1  I 

I - ! 

5  I  0 

100.0  I  .0 

21.7  I  .0 

15.6  I  .0 


SOMEWHAT  USEFUL 


INDIFFERENT 


COLUMN 

TOTAL 


I 
I 
I 
I 

- 1 . 

3  I 


50.0 

13.0 

9.4 


2 

50.0 

8.7 

6.3 


23 

71.9 


3 

50.0 

33.3 

9.4 


-I- 

I 

7 

I 

2 

I 

77.8 

I 

22.2 

I 

30.4 

I 

22.2 

I 

-  T  - 

21.9 

I 

6.3 

- 1  - 

I 

6 

I 

2 

I 

75.0 

I 

25.0 

I 

26. 1 

I 

22.2 

I 

18.8 

I 

6.3 

-I- 

2 

50.0 

22.2 

6.3 


9 

28.  1 


RDW 

TOTAL 


5 

15.6 


6 

18.8 


28 


25 


12 


32 

100.0 


8  OUT  DF  10  (  80.0%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  1.125 

RAW  CHI  SQUARE  =  4.51744  WITH  4  DEGREES  DF  FREEDOM.  SIGNIFICANCE  »  .3405 

CRAMER  ' S  V  =  .37573 

CONTINGENCY  COEFFICIENT  =  .35172 

LAMBDA  (ASYMMETRIC)  =  .04348  WITH  USRCDM  DEPENDENT.  =  .00000  WITH  USER  DEPENDENT 

LAMBOA  (SYMMETRIC)  =  .03125 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .05612  WITH  USRCDM  DEPENDENT.  =  .14804  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .08139 

.  2070 
.2070 


.  13189. 
.  14844. 


SIGNIFICANCE  = 
SIGNIFICANCE  = 


KENDALL'S  TAU  B  = 

KENDALL'S  TAU  C  = 

GAMMA  =  .22619 

SOMERS'S  D  (ASYMMETRIC)  =  .18357  WITH  USRCDM 

SOMERS'S  D  (SYMMETRIC)  =  .12500 

ETA  =  .16681  WITH  USRCOM  DEPENDENT. 

PEARSON'S  R  =  .16681  SIGNIFICANCE  =  .1808 


OEPENOENT . 


.37573  WITH  USER 


.09476  WITH  USER 


DEPENOENT. 


OEPENDENT . 


*  *  * 

1  OF  1 


DEPENOENT . 


******************  CRDSSTABULATION  OF  ****************** 
CHOCDM  USER  CHODSES  COM  LANG  BY  USER  USER  TYPE 

*************************************************  PAGE  1  DF  1 

USER 

COUNT  I 

ROW  PCT  I INTERMEO  ENO-USER  ROW 


COL 

TOT 

PCT 

PCT 

1 1  ARY 

I  1 

I 

_  T  - 

2 

I 

_  T 

TOTAL 

LMULUM 

X  - 

1 

1 

1 

I 

0 

I 

1 

I 

1 

NOT  USEFUL 

I 

.0 

I 

100.0 

I 

3.  1 

I 

.0 

I 

11.1 

I 

I 

.0 

I 

3.  1 

I 

I- 

-I- 

-I 

2 

I 

1 

I 

0 

I 

1 

SOMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3. 1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

I- 

-I- 

-I 

3 

I 

0 

I 

6 

I 

6 

INO I FFERENT 

I 

.0 

I 

100.0 

I 

18.8 

I 

.0 

I 

66.7 

I 

I 

.0 

I 

18.8 

I 

I- 

-I- 

-I 

4 

I 

1 1 

I 

1 

I 

12 

VERY  USEFUL 

I 

91.7 

I 

8.3 

I 

37.5 

I 

47.8 

I 

11.1 

I 

I 

34.4 

I 

3.  1 

I 

- 

I- 

-I- 

-I 

5 

I 

1  1 

I 

1 

I 

12 

ESSENTIAL 

I 

91.7 

I 

8.3 

I 

37.5 

I 

47.8 

I 

11.1 

I 

I 

34.4 

I 

3.  1 

I 

- 

I- 

-I" 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28.  1 

100.0 

8  OUT  OF  10  (  80.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTEO  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .281 

RAW  CHI  SQUARE  =  22.93076  WITH  4  OEGREES  OF  FREEOOM .  SIGNIFICANCE  =  .0001 

CRAMER  9 S  V  =  .84651 

CONTINGENCY  COEFFICIENT  =  .64610 

LAMBDA  (ASYMMETRIC)  =  .25000  WITH  CHOCOM  OEPENOENT .  =  .77778  WITH  USER  OEPENOENT 


LAMBOA  (SYMMETRIC)  =  .41379 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .29935 

WITH  CHOCDM 

OEPENOENT. 

_ 

.63791 

WITH  USER 

DEPENDENT 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .40748 

KENDALL ' S  TAU  B  =  -.54691.  SIGNIFICANCE  = 

.0005 

KENDALL  * S  TAU  C  =  -.57422.  SIGNIFICANCE  = 

GAMMA  =  -.79459 

.0005 

SDMERS ' S  0  (ASYMMETRIC)  =  -.71014  WITH  CHOCOM 

SOMERS'S  0  (SYMMETRIC)  =  -.52878 

DEPENOENT. 

— 

-.42120  WITH 

USER 

OEPENDENT. 

ETA  =  .58509  WITH  CHOCDM  DEPENDENT. 

PEARSON'S  R  =  -.58509  SIGNIFICANCE  =  .0002 

=  .84651 

WITH  USER 

OEPENDENT. 

*  *  *  *  * 


*****  *  * 


*  *  * 


********** 


******** 


**  CRDSST  ABULAT  IDN  DF 
LEVEL  USER  CHODSE  LEVEL  DF  EXP  BY  USER  USER  TYPE 

*************************************************  PAGE 

USER 

CDUNT  I 

RDW  PCT  IINTERMED  END-USER  RDW 


CDL  PCT 
TOT  PCT 

1 1  ARY 

I  1 

I 

2 

I 

TOTAL 

LEVEL  - 

2 

I 

2 

I 

0 

I 

2 

SDMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

6.3 

I 

8.7 

I 

.0 

I 

I 

6.3 

I 

.0 

I 

-I- 

-I- 

-I 

3 

I 

5 

I 

3 

I 

8 

INDIFFERENT 

I 

62.5 

I 

37.5 

I 

25.0 

I 

21.7 

I 

33.3 

I 

I 

15.6 

I 

9.4 

I 

-I- 

-I- 

-I 

4 

I 

11 

I 

1 

I 

12 

VERY  USEFUL 

I 

91.7 

I 

8.3 

I 

37.5 

I 

47.8 

I 

11.1 

I 

I 

34.4 

I 

3.  1 

I 

-I- 

-I- 

-I 

5 

I 

5 

I 

5 

I 

10 

ESSENTIAL 

I 

50.0 

I 

50.0 

I 

31.3 

I 

21.7 

I 

55.6 

I 

I 

15.6 

I 

15.6 

I 

-I  - 

-I- 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28. 1 

100.0 

5  OUT  OF  8  (  62.5%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  =  5.82287  WITH  3  DEGREES  OF  FREEDDM.  SIGNIFICANCE  =  .1206 

CRAMER'S  V  =  .42657 

CONTINGENCY  COEFFICIENT  =  .39237 

LAMBDA  (ASYMMETRIC)  =  .20000  WITH  LEVEL  DEPENDENT.  =  .OOOOO  WITH  USER 

LAMBOA  (SYMMETRIC)  =  .13793 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .08358  WITH  LEVEL  DEPENDENT. 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .11333 


5.0. 


OEPENDENT . 


. 17600  WITH  USER 


KENDALL'S  TAU  B  =  .18419. 

KENDALL'S  TAU  C  =  .19531. 

GAMMA  *  .32051 

SDMERS'S  D  (ASYMMETRIC)  = 

SOMERS'S  D  (SYMMETRIC)  = 

ETA  =  . 19807  WITH  LEVEL 

PEARSDN'S  R  =  .19807  SIGNIFICANCE  = 


SIGNIFICANCE  = 
SIGNIFICANCE  = 

.24155  WITH  LEVEL 
17762 

DEPENDENT. 

.  1386 


.  1349 
.  1349 

DEPENDENT. 


.42657  WITH  USER 


. 14045  WITH  USER 


DEPENDENT. 


DEPENDENT . 


1  DF  1 


OEPENOENT . 


******************  CRDSSTABULATIDN  DF  ************** 
NATLAN  NATURAL  LANGUAGE  BY  USER  USER  TYPE 

*************************************************  page 

USER 

CDUNT  I 

RDW  PCT  1 1 NTERMED  END-USER  RDW 
CDL  PCT  1 1  ARY  TDTAL 


TDT  PCT 

I 

1 

I 

2 

I 

NATLAN  - 

-I- 

-I- 

-I 

1 

I 

3 

I 

2 

I 

5 

NDT  USEFUL 

I 

60.0 

I 

40.0 

I 

15.6 

I 

13.0 

I 

22.2 

I 

I 

9.4 

I 

6.3 

I 

-I- 

-I- 

-I 

2 

I 

3 

I 

1 

I 

4 

SDMEWHAT  USEFUL 

I 

75.0 

I 

25.0 

I 

12.5 

I 

13.0 

I 

11.1 

I 

I 

9.4 

I 

3.  1 

I 

-I- 

-I- 

-I 

3 

I 

6 

I 

2 

I 

8 

INDIFFERENT 

I 

75.0 

I 

25.0 

I 

25.0 

I 

26.  1 

I 

22.2 

I 

I 

18.8 

I 

6.3 

I 

-I- 

-I- 

-I 

4 

I 

8 

I 

1 

I 

9 

VERY  USEFUL 

I 

88.9 

I 

11.1 

I 

28.  1 

I 

34.8 

I 

11.1 

I 

I 

25.0 

I 

3. 1 

I 

-I- 

-I- 

-I 

5 

I 

3 

I 

3 

I 

6 

ESSENTIAL 

I 

50.0 

I 

50.0 

I 

18.8 

I 

13.0 

I 

33.3 

I 

I 

9.4 

I 

9.4 

I 

-I- 

-I- 

-I 

CDLUMN 

23 

9 

32 

TDTAL 

71.9 

28.  1 

100.0 

.5386 


.OOOOO  WITH  USER 


8  DUT  DF  10  (  80.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 
MINIMUM  EXPECTED  CELL  FREQUENCY  =  1.125 

RAW  CHI  SQUARE  =  3.11583  WITH  4  DEGREES  DF  FREEDDM.  SIGNIFICANCE  = 

CRAMER ' S  V  =  .31204 

CONTINGENCY  COEFFICIENT  =  .29788 

LAMBDA  (ASYMMETRIC)  =  .08696  WITH  NATLAN  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .06250 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .03192  WITH  NATLAN  DEPENDENT. 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .04629 

KENDALL'S  TAU  B  *  .01735.  SIGNIFICANCE  = 

KENDALL ' S  TAU  C  =  .01953.  SIGNIFICANCE  = 

GAMMA  =  .02959 

SDMERS ' S  D  (ASYMMETRIC)  =  .02415  WITH  NATLAN 

SDMERS ' S  D  (SYMMETRIC)  =  .01645 

ETA  =  .00164  WITH  NATLAN  DEPENDENT. 

PEARSDN'S  R  =  .00165  SIGNIFICANCE  =  .4964 


DEPENDENT. 


.08419  WITH  USER 


.4572 
.  4572 

DEPENDENT. 


.31204  WITH  USER 


.01247  WITH  USER 


DEPENDENT. 


DEPENDENT. 


1  OF  1 


DEPENDENT. 


************  CROSSTABULATION  DF  **************t 
SPELLING  ERRORS  BY  USER  USER  TYPE 

*******************************************  PAGE 

USER 

COUNT  I 

ROW  PCT  I INTERMED  END-USER  ROW 


COL 

TOT 

PCT 

PCT 

1 1  ARY 

I  1 

I 

_  T  _ 

2 

I 

T 

TDTAL 

orLtKK  - - 

i 

1 

1 

1 

I 

3 

I 

3 

I 

6 

NDT  USEFUL 

I 

50.0 

I 

50.0 

I 

18.8 

I 

13.0 

I 

33.3 

I 

I 

9.4 

I 

9.4 

I 

I- 

-I- 

-I 

2 

I 

4 

I 

1 

I 

5 

SOMEWHAT  USEFUL 

I 

80.0 

I 

20.0 

I 

15.6 

I 

17.4 

I 

11.1 

I 

I 

12.5 

I 

3.  1 

I 

I- 

-I- 

-I 

3 

I 

3 

I 

1 

I 

4 

INDIFFERENT 

I 

75.0 

I 

25.0 

I 

12.5 

I 

13.0 

I 

11.1 

I 

I 

9.4 

I 

3.  1 

I 

I- 

-I- 

-I 

4 

I 

6 

I 

1 

I 

7 

VERY  USEFUL 

I 

85.7 

I 

14.3 

I 

21.9 

I 

26.  1 

I 

11.1 

I 

I 

18.8 

I 

3.  1 

I 

- 

I- 

-I* 

-I 

5 

I 

7 

I 

3 

I 

10 

ESSENTIAL 

I 

70.0 

I 

30.0 

I 

31.3 

I 

30.4 

I 

33.3 

I 

I 

21.9 

I 

9.4 

I 

- 

I- 

"I" 

-I 

CDLUMN 

23 

9 

32 

TOTAL 

71.9 

28. 1 

100.0 

8  DUT  DF  10  (  80.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  1.125 

CELL  FREQUENCY 

LESS 

THAN 

5.0. 

RAW  CHI  SQUARE  =  2.28351  WITH  4  DEGREES  DF  FREEOOM . 

CRAMER'S  V  =  .26713 

CONTINGENCY  COEFFICIENT  =  .25808 

SIGNIFICANCE  = 

.6838 

LAMBDA  (ASYMMETRIC)  *  .00000  WITH  SPLERR  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  . OOOOO 

=  . OOOOO 

WITH 

USER 

DEPENDENT . 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .02249  WITH  SPLERR 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .03257 

KENDALL'S  TAU  B  =  -.09743.  SIGNIFICANCE  =  .2733 

DEPENDENT. 

05904  WITH  USER 

KENOALL  9 S  TAU  C  =  -.10938.  SIGNIFICANCE  =  .2733 

GAMMA  =  -.17073 

SOMERS ' S  D  (ASYMMETRIC)  =  -.13527  WITH  SPLERR  OEPENDENT .  =  -.07018  WITH  USER  DEPENDENT. 

SOMERS' S  0  (SYMMETRIC)  =  -.09241 

ETA  =  .12954  WITH  SPLERR  DEPENOENT.  =  .26713  WITH  USER  DEPENOENT. 

PEARSON'S  R  =  -.12954  SIGNIFICANCE  =  .2399 


****** 

SPLERR 

****** 


*  *  * 

1  OF  1 


DEPENOENT. 


******************  CROSSTABULATION  OF  *************** 
QUEST  QUEST  FOR  SEARCH  STRAT  BY  USER  USER  TYPE 

*************************************************  PAGE 


COUNT 
ROW  PCT 
COL  PCT 
TOT  PCT 


USER 

I 

I INTERMEO 

I I  ARY 

I  1 


ENO-USER 


QUEST 


1 


NOT  USEFUL 


SOMEWHAT  USEFUL 


INDIFFERENT 


VERY  USEFUL 


ESSENTIAL 


7 

77.8 
30.4 

21.9 


3 

60.0 

13.0 

9.4 


4 

80.0 

17.4 

12.5 


6 

85.7 
26.  1 

18.8 


3 

50.0 

13.0 

9.4 


2 

22.2 

22.2 

6.3 


I  2 

I  40.0 
I  22.2 
I  6.3 


I  1 

I  20.0 
I  11.1 
I  3.1 

- I - 


I  1 

I  14.3 
I  11.1 
I  3.1 


I  3 

I  50.0 
I  33.3 
I  9.4 
*  I - 


COLUMN 

TOTAL 


23 
71 .9 


9 

28 . 1 


ROW 

TOTAL 


9 

28. 1 


5 

15.6 


5 

15.6 


7 

21  .9 


6 

18.8 


32 

100.0 


.OOOOO  WITH  USER 


OEPENOENT . 


8  OUT  OF  10  (  80.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTEO  CELL  FREQUENCY  LESS  THAN  5.0. 
MINIMUM  EXPECTEO  CELL  FREQUENCY  =  1.406 

RAW  CHI  SQUARE  =  2.75071  WITH  4  OEGREES  OF  FREEOOM.  SIGNIFICANCE  =  .6004 

CRAMER ' S  V  =  .29319 

CONTINGENCY  COEFFICIENT  =  .28135 

LAMBOA  (ASYMMETRIC)  =  .04348  WITH  QUEST  DEPENDENT. 

LAMBOA  (SYMMETRIC)  =  .03125 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .02661  WITH  QUEST 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .03870 

KENOALL'S  TAU  B  =  .09682.  SIGNIFICANCE  = 

KENOALL'S  TAU  C  =  .10938.  SIGNIFICANCE  = 

GAMMA  =  . 16667 

SOMERS ' S  0  (ASYMMETRIC)  =  .13527  WITH  QUEST 

SOMERS'S  0  (SYMMETRIC)  =  .09165 

ETA  =  .09881  WITH  QUEST  DEPENOENT. 

PEARSON'S  R  =  .09881  SIGNIFICANCE  =  .2953 


OEPENOENT. 


.07090  WITH  USER 


.2740 

.2740 

OEPENOENT. 


.29319  WITH  USER 


.06931  WITH  USER 


OEPENOENT. 


OEPENOENT. 


*  *  * 

1  OF  1 


OEPENOENT. 


*  *  *  *  * 


****** 


************ 


******************  CROSSTABULATION  OF 

RELTER  RELATED  TERMS  BY  USER  USER  TYPE 

*************************************************  PAGE 


COUNT 
ROW  PCT 
CDL  PCT 


USER 

I 

1 1 NTERMED  END-USER 

I I  ARY 


ROW 

TOTAL 


RELTER  - 

1 

I 

1 

I 

1 

I 

2 

NDT  USEFUL 

I 

50.0 

I 

50.0 

I 

6.3 

I 

4.3 

I 

11.1 

I 

I 

3.  1 

I 

3. 1 

I 

-I  - 

*1- 

--I 

'  2 

I 

0 

I 

1 

I 

1 

SOMEWHAT  USEFUL 

I 

.0 

I 

100.0 

I 

3. 1 

I 

.0 

I 

11.1 

I 

I 

.0 

I 

3.  1 

I 

-I- 

-I- 

--I 

3 

I 

3 

I 

2 

I 

5 

INDIFFERENT 

I 

60.0 

I 

40.0 

I 

15.6 

I 

13.0 

I 

22.2 

I 

I 

9.4 

I 

6.3 

I 

-I- 

-I- 

--I 

4 

I 

9 

I 

2 

I 

1  1 

VERY  USEFUL 

I 

81.8 

I 

18.2 

I 

34.4 

I 

39.  1 

I 

22.2 

I 

I 

28. 1 

I 

6.3 

I 

-I- 

-I- 

5 

I 

10 

I 

3 

I 

13 

ESSENTIAL 

I 

76.9 

I 

23.  1 

I 

40.6 

I 

43.5 

I 

33.3 

I 

I 

31.3 

I 

9.4 

I 

-I- 

-I- 

--I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28.  1 

100.0 

8  OUT  OF  10 

( 

80.0%) 

OF 

THE  i 

YALID  CELLS 

.281 

4  DEGREES  OF  FREEDOM. 


MINIMUM  EXPECTED  CELL  FREQUENCY  “ 

RAW  CHI  SQUARE  *  4.07965  WITH 

CRAMER ' S  V  =  .35706 

CONTINGENCY  COEFFICIENT  =  .33626 

LAMBDA  (ASYMMETRIC)  =  . OOOOO  WITH  RELTER 

LAMBDA  (SYMMETRIC)  =  .03571 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .04845  WITH  RELTER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .06658 

KENDALL #  S  TAU  B  =  -.19264.  SIGNIFICANCE  = 

KENDALL #  S  TAU  C  =  -.20313.  SIGNIFICANCE  = 

GAMMA  =  -.34211 

SOMERS'S  D  (ASYMMETRIC)  *  -.25121  WITH  RELTER 

SOMERS'S  D  (SYMMETRIC)  *  -.18605 

ETA  =  .24867  WITH  RELTER  DEPENDENT. 

PEARSON'S  R  =  -.24867  SIGNIFICANCE  =  .0850 


SIGNIFICANCE  *  .3953 


DEPENDENT. 


.  1239 
.  1239 


11111  WITH  USER 


DEPENDENT. 


OE  PENDENT . 


. 10639  WITH  USER 


OEPENOENT . 


.35706  WITH  USER 


■. 14773  WITH  USER 


DEPENDENT. 


DEPENDENT . 


1  OF  1 


DEPENDENT. 


*  *  *  *  * 
FEEDBK 
*  *  *  *  * 


FEEDBK  ON  SEAR  STRAT 
*********** 


CROSSTABULATION 

BY  USER 

*************  +  + 


OF  *  *  *  * 
USER  TYPE 
******* 


****** 


*  *  PAGE 


FEEDBK 


COUNT 
ROW  PCT 
COL  PCT 
TDT  PCT 


1 


NOT  USEFUL 


SOMEWHAT  USEFUL 


INDIFFERENT 


VERY  USEFUL 


ESSENTIAL 


USER 

I 

I INTERMED  END-USER 

I I  ARY 

I  II  2 

I - ! - 

2  I  1 

66.7  I  33.3 
8.7  I  11.1 
6.3  I  3.1 


ROW 

TOTAL 


3 

9.4 


3  I 

0 

I 

100.0  I 

.0 

I 

13.0  I 

.0 

I 

9.4  I 

.0 

I 

2 

33.3 

8.7 

6.3 


10 

83.3 
43.5 

31 .3 


4 

66.7 

44.4 

12.5 


2 

16.7 

22.2 

6.3 


-I- 

I 

6 

I 

2 

I 

I 

75.0 

I 

25.0 

I 

I 

26.  1 

I 

22.2 

I 

I 

18.8 

I 

6.3 

I 

-I- 

COLUMN 

TOTAL 


23 

71.9 


9 

28.  1 


3 

9.4 


6 

18.8 


12 

37.5 


8 

25.0 


32 

100.0 


minimu^expected^eJl^reouency7^  VA844  CELLS  HAVE  EXPECTED  CELL  FRE0UENCY  less  5  0- 

rDAMPDicSwUARE  =  6.44122  WITH  4  DEGREES  OF  FREEDOM.  SIGNIFICANCE  =  .1685 

CKAMtK  b  V  =  . 44865 

CONTINGENCY  COEFFICIENT  =  .40934 

LAMBDA  (ASYMMETRIC)  =  . lOOOO  WITH  FEEDBK  DEPENDENT.  =  22222  WITH  ikfd  nPDPMHPMT 

LAMBDA  (SYMMETRIC)  =  .13793  .22222  WITH  USER  DEPENDENT. 

.07171  WITH  FEEDBK  DEPENDENT.  =  .17768  WITH  USER 


UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  = 
UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .10218 

KENDALL'S  TAU  B  =  -.10326.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  -.11328.  SIGNIFICANCE  = 

GAMMA  =  -.17576 

SOMERS'S  D  (ASYMMETRIC)  =  -.14010  WITH  FEEDBK 

SOMERS'S  D  (SYMMETRIC)  =  -.09864 

ETA  =  .07648  WITH  FEEDBK  DEPENDENT. 

PEARSON'S  R  =  -.07648  SIGNIFICANCE  =  .3387 


.  2636 
.  2636 

DEPENDENT.  =  -.07612  WITH  USER  DEPENDENT. 

=  .44865  WITH  USER  DEPENDENT. 


*  *  *  * 

1  DF  1 


DEPENDENT. 


******** 


************ 


CRDSST  ABU 


SELECT 
*****  * 


SYS  DR  USER  SELECTS  DB 
************ 


LATIDN  DF  **** 
BY  USER  USER  TYPE 

************** 


PAGE 


USER 

CDUNT  I 

RDW  PCT  I I NTERMED  END-USER 

CDL  PCT  1 1  ARY 

TDT  PCT  I  II  2 


SELECT  - 

1 

I 

1 

I 

1 

NDT  USEFUL 

I 

50.0 

I 

50.0 

I 

4.3 

I 

11.1 

I 

3. 1 

I 

3. 1 

-I- 

-I- 

2 

I 

2 

I 

0 

SDMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

8.7 

I 

.0 

I 

6.3 

I 

.0 

-I- 

-I- 

3 

I 

6 

I 

3 

INDIFFERENT 

I 

66.7 

I 

33.3 

I 

26.  1 

I 

33.3 

I 

18.8 

I 

9.4 

-I- 

-I- 

4 

I 

7 

I 

2 

VERY  USEFUL 

I 

77.8 

I 

22.2 

I 

30.4 

I 

22.2 

I 

21.9 

I 

6.3 

-I- 

-I- 

5 

I 

7 

I 

3 

ESSENTIAL 

I 

70.0 

I 

30.0 

I 

30.4 

I 

33.3 

I 

21.9 

I 

9.4 

-I- 

-I- 

CDLUMN 

23 

9 

TOTAL 

71.9 

28.  1 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 


RDW 

TDTAL 


2 

6.3 


2 

6.3 


9 

28.  1 


9 

28. 1 


10 
31 .3 


32 

100.0 


DEPENDENT. 


7  DUT  DF  10  (  70.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0 
MINIMUM  EXPECTED  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  *  1.54933  WITH  4  DEGREES  DF  FREEDDM.  SIGNIFICANCE  *  .8179 

CRAMER'S  V  =  .22004 

CONTINGENCY  COEFFICIENT  =  .21490 

LAMBDA  (ASYMMETRIC)  -  .OOOOO  WITH  SELECT  DEPENDENT.  .  =  . OOOOO  WITH  USER 

LAMBDA  (SYMMETRIC)  =  .OOOOO 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .02242  WITH  SELECT 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .03163 

KENDALL'S  TAU  B  =  -.01074.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  -.01172.  SIGNIFICANCE  = 

GAMMA  =  -.01961 

SDMERS'S  D  (ASYMMETRIC)  =  -.01449  WITH  SELECT 

SDMERS'S  D  (SYMMETRIC)  *  -.01027 

ETA  =  .02828  WITH  SELECT  DEPENDENT. 

PEARSDN'S  R  *  -.02828  SIGNIFICANCE  =  .4389 


DEPENDENT. 


.05372  WITH  USER 


.4739 

.4739 

DEPENDENT. 


.22004  WITH  USER 


- .00796  WITH  USER 


DEPENDENT. 


DEPENDENT. 


1  DF  1 


DEPENDENT. 


*  *  *  *  *  * 


************ 


***************** 


SYSSEL 


SYSTEM  SELECTS 
******** 


R  0 
DB 


S  S  T  A  B  U  L 


A  T  I  0  N 
BY  USER 


0  F 


USER  TYPE 


*  *  *  PAGE 


CDUNT 
ROW  PCT 
COL  PCT 
TOT  PCT 


USER 

I 

I INTERMEO 

I I  ARY 

I  1 


ENO-USER 


ROW 

TOTAL 


SYSSEL 


1 


NOT  USEFUL 


SOMEWHAT  USEFUL 


12 

I 

5 

I 

17 

70.6 

I 

29.4 

I 

53.  1 

52.2 

I 

55.6 

I 

37.5 

I 

15.6 

I 

-I- 

-I 

1 

I 

1 

I 

2 

50.0 

I 

50.0 

I 

6.3 

4.3 

I 

11.1 

I 

3.  1 

I 

3.  1 

I 

3 

I 

4 

I 

1 

I  5 

INDIFFERENT 

I 

80.0 

I 

20.0 

I  15.6 

I 

17.4 

I 

11.1 

I 

I 

12.5 

I 

3.  1 

I 

-I- 

- 1  -- 

--I 

4 

I 

5 

I 

1 

I  6 

VERY  USEFUL 

I 

83.3 

I 

16.7 

I  18.8 

I 

21.7 

I 

11.1 

I 

I 

15.6 

I 

3.  1 

I 

-I- 

-I-- 

--I 

5 

I 

1 

I 

1 

I  2 

ESSENTIAL 

I 

50.0 

I 

50.0 

I  6.3 

I 

4.3 

I 

11.1 

I 

I 

3.  1 

I 

3.  1 

I 

-I  - 

-I-- 

--I 

COLUMN 

23 

9 

32 

TDTAL 

71.9 

28. 1 

100.0 

9  DUT  OF 

10 

( 

90 . 0% ) 

OF 

THE  ' 

VALIO  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0 

MINIMUM  EXPECTEO  CELL 

FREQUENCY 

.563 

1 .51376  WITH 


RAW  CHI  SQUARE  = 

CRAMER'S  V  =  .21750 

CONTINGENCY  COEFFICIENT  =  .21253 

LAMBOA  (ASYMMETRIC)  =  . OOOOO  WITH  SYSSEL 

LAMBDA  (SYMMETRIC)  =  .OOOOO 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  *  .01787  WITH  SYSSEL 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .02445 

KENDALL'S  TAU  B  =  -.03047.  SIGNIFICANCE  - 

KENOALL'S  TAU  C  =  -.03125.  SIGNIFICANCE  = 

GAMMA  =  -.05882 

SOMERS'S  0  (ASYMMETRIC)  =  -.03865  WITH  SYSSEL 

SOMERS'S  D  (SYMMETRIC)  =  -.02963 

ETA  =  .03409  WITH  SYSSEL  OEPENDENT . 

PEARSON'S  R  =  -.03409  SIGNIFICANCE  =  .4265 


4  OEGREES  OF  FREEDOM.  SIGNIFICANCE  =  .8242 


OEPENDENT. 


.4274 

.4274 


.00000  WITH  USER 


OEPENDENT . 


OEPENOENT . 


.03869  WITH  USER 


DEPENOENT. 


.21750  WITH  USER 


-.02402  WITH  USER 


DEPENDENT. 


OEPENOENT. 


1  DF  1 


OEPENOENT. 


****************** 


CRDSSTABULATIDN  DF  ************** 
PROCES  DISPLAYS  PRDCESS  BY  USER  USER  TYPE 

*************************************************  PAGE 


USER 

CDUNT  I 

ROW  PC T  I INTERMED  END-USER  RDW 
CDL  PCT  I I ARY  TOTAL 


TDT  PCT 

I 

1 

I 

2 

I 

KUL to  - 

1 

I 

2 

I 

1 

I 

3 

NDT  USEFUL 

I 

66.7 

I 

33.3 

I 

9.4 

I 

8.7 

I 

11.1 

I 

I 

6.3 

I 

3.  1 

I 

-I- 

-I- 

-I 

2 

I 

1 

I 

0 

I 

1 

SOMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I- 

-I- 

-I 

3 

I 

3 

I 

2 

I 

5 

INDIFFERENT 

I 

60.0 

I 

40.0 

I 

15.6 

I 

13.0 

I 

22.2 

I 

I 

9.4 

I 

6.3 

I 

-I- 

-I- 

-I 

4 

I 

6 

I 

2 

I 

8 

VERY  USEFUL 

I 

75.0 

I 

25.0 

I 

25.0 

I 

26.  1 

I 

22.2 

I 

I 

18.8 

I 

6.3 

I 

-I- 

-I- 

-I 

5 

I 

1  1 

I 

4 

I 

15 

ESSENTIAL 

I 

73.3 

I 

26.7 

I 

46.9 

I 

47.8 

I 

44.4 

I 

I 

34.4 

I 

12.5 

I 

-I- 

-I- 

-I 

CDLUMN 

23 

9 

32 

TDTAL 

71.9 

28.  1 

100.0 

8  DUT  DF 


10  (  80.0%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 


.281 

4  DEGREES  OF 


MINIMUM  EXPECTED  CELL  FREQUENCY  * 

RAW  CHI  SQUARE  *  .83478  WITH 

CRAMER'S  V  =  . 16151 

CONTINGENCY  COEFFICIENT  =  .15945 

LAMBDA  (ASYMMETRIC)  =  . OOOOO  WITH  PRDCES 

LAMBOA  (SYMMETRIC)  =  .OOOOO 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .01277  WITH  PRDCES 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  *  .01762 

KENDALL'S  TAU  B  =  -.04087.  SIGNIFICANCE  = 

KENOALL'S  TAU  C  =  -.04297.  SIGNIFICANCE  = 

GAMMA  =  -.07692 

SOMERS'S  D  (ASYMMETRIC)  =  -.05314  WITH  PROCES 

SDMERS'S  D  (SYMMETRIC)  =  -.03950 

ETA  =  .03958  WITH  PRDCES  DEPENDENT. 

PEARSDN'S  R  =  -.03958  SIGNIFICANCE  =  .4148 


FREEDOM.  SIGNIFICANCE  * 


.9337 


DEPENDENT . 


.4028 

.4028 


.00000  WITH  USER 


DEPENDENT. 


DEPENDENT. 


.02841  WITH  USER 


DEPENDENT . 


16151  WITH  USER 


-.03143  WITH  USER 


DEPENDENT. 


DEPENDENT. 


*  *  * 

1  OF  1 


DEPENDENT. 


*  *  *  *  * 


*  *  *  * 


******* 


***************** 


*  CRDSSTABULATION  DF  * 

RANKS  RANKS  BY  RELEVANCY  BY  USER  USER  TYPE 

*************************************************  PAGE 

USER 

COUNT  I 

ROW  PCT  I INTERMEO  END-USER  ROW 
COL  PCT  1 1  ARY  TOTAL 


TOT  PCT 

I 

T  _ 

1 

I 

T 

2 

I 

T 

ANK  j 

i 

- 1  - 

1 

i 

I 

2 

I 

4 

I 

6 

NOT  USEFUL 

I 

33.3 

I 

66.7 

I 

18.8 

I 

8.7 

I 

44.4 

I 

I 

6.3 

I 

12.5 

I 

-I- 

-I  - 

-I 

2 

I 

1 

I 

0 

I 

1 

SOMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I- 

-I- 

-I 

3 

I 

5 

I 

1 

I 

6 

INDIFFERENT 

I 

83.3 

I 

16.7 

I 

18.8 

I 

21.7 

I 

11.1 

I 

I 

15.6 

I 

3.  1 

I 

-I- 

-I- 

-I 

4 

I 

8 

I 

1 

I 

9 

VERY  USEFUL 

I 

88.9 

I 

11.1 

I 

28. 1 

I 

34.8 

I 

11.1 

I 

I 

25.0 

I 

3.  1 

I 

-I- 

-I- 

-I 

5 

I 

7 

I 

3 

I 

10 

ESSENTIAL 

I 

70.0 

I 

30.0 

I 

31.3 

I 

30.4 

I 

33.3 

I 

I 

21 .9 

I 

9.4 

I 

-I- 

-I  - 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28.  1 

100.0 

8  DUT  DF  10  (  80.0%)  DF  THE  VALIO  CELLS  HAVE  EXPECTED 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  .281 

CELL  FREQUENCY 

LESS 

THAN 

5.0. 

RAW  CHI  SQUARE  =  6.49619  WITH  4  OEGREES  DF  FREEDOM. 

CRAMER'S  V  =  .45056 

CONTINGENCY  COEFFICIENT  =  .41079 

SIGNIFICANCE  = 

.  1650 

LAMBDA  (ASYMMETRIC)  =  .09091  WITH  RANKS  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .12903 

=  .22222 

WITH 

USER 

DEPENDENT. 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .06956  WITH  RANKS 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .09881 

KENDALL'S  TAU  B  =  -.16649.  SIGNIFICANCE  =  .1542 

DEPENDENT. 

. 17050  WITH  USER 

KENOALL'S  TAU  C  =  -.18359.  SIGNIFICANCE  =  .1542 

GAMMA  =  -.28485 

SOMERS ' S  0  (ASYMMETRIC)  =  -.22705  WITH  RANKS  OEPENOENT .  =  -.12208  WITH  USER  DEPENDENT. 

SOMERS'S  0  (SYMMETRIC)  =  -.15878 

ETA  =  .26618  WITH  RANKS  DEPENDENT.  =  .45056  WITH  USER  OEPENOENT. 

PEARSON'S  R  =  -.26618  SIGNIFICANCE  =  .0704 


1  OF  1 


OEPENOENT. 


******************  CROSSTABULATION  OF  ******** 
EXPERR  EXPLAINS  ERRDRS  BY  USER  USER  TYPE 

t**********************************^^^^^^^^^^^ 


USER 

CDUNT  I 


RDW 

CDL 

TDT 

EXPERR  _ * 

PCT 

PCT 

PCT 

I INTERMED 

1 1 ARY 

I  1  I 

END-USER 

2  I 

RDW 

TDTAL 

1 

I 

1 

I 

0 

I 

1 

NOT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3. 1 

I 

.0 

I 

- 1  - 

-I 

2 

I 

1 

I 

0 

I 

1 

SDMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

■I- 

-I 

-I 

3 

I 

4 

I 

1 

I 

5 

INDIFFERENT 

I 

80.0 

I 

20.0 

I 

15.6 

I 

17.4 

I 

11.1 

I 

I 

12.5 

I 

3. 1 

I 

I  - 

-I 

-I 

4 

I 

10 

I 

2 

I 

12 

VERY  USEFUL 

I 

83.3 

I 

16.7 

I 

37.5 

I 

43.5 

I 

22.2 

I 

I 

31.3 

I 

6.3 

I 

- 

I- 

-I 

-I 

5 

I 

7 

I 

6 

I 

13 

ESSENTIAL 

I 

53.8 

I 

46.2 

I 

40.6 

I 

30.4 

I 

66.7 

I 

I 

21.9 

I 

18.8 

I 

- 

I- 

-I- 

-I 

CDLUMN 

23 

9 

32 

TDTAL 

71.9 

28.  1 

100.0 

*  *  * 

PAGE 


SIGNIFICANCE  = 


.4315 


.00000  WITH  USER 


8  OUT  DF  10  (  80.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 
MINIMUM  EXPECTED  CELL  FREQUENCY  =  .281 

RAW  CHI  SQUARE  =  3.81558  WITH  4  DEGREES  DF  FREEDOM. 

CRAMER'S  V  *  .34531 

CONTINGENCY  COEFFICIENT  =  .32640 

LAMBDA  (ASYMMETRIC)  =  .15789  WITH  EXPERR  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  ,10714 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  ,05369  WITH  EXPERR 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .07260 

KENDALL'S  TAU  B  =  .29691.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  .30859.  SIGNIFICANCE  = 

GAMMA  =  .56028 

SOMERS'S  D  (ASYMMETRIC)  =  .38164  WITH  EXPERR 

SOMERS'S  D  (SYMMETRIC)  =  .28780 

ETA  »  .29484  WITH  EXPERR  DEPENDENT. 

PEARSON'S  R  =  .29484  SIGNIFICANCE  =  .0507 


DEPENDENT. 


DEPENDENT . 


1  1209  WITH  USER 


.0386 

.0386 

DEPENDENT. 


.34531  WITH  USER 


.23099  WITH  USER 


DEPENOENT. 


DEPENDENT . 


*  *  * 

1  DF  1 


OE  PENDENT . 


******************  CROSSTABULATION  OF  ************** 
HELP  HELP  FEATURE  BY  USER  USER  TYPE 

*************************************************  pAGE 

USER 

COUNT  I 

ROW  PCT  IINTERMEO  ENO-USER  ROW 
COL  PCT  1 1  ARY  *  TOTAL 

TOT  PCT  I  II  21 


:i_r  - 

1 

I 

1 

I 

0 

I 

1 

NOT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I- 

-I- 

-I 

2 

I 

1 

I 

0 

I 

1 

SOMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3. 1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I- 

-I- 

-I 

3 

I 

3 

I 

1 

I 

4 

INO I F  FERENT 

I 

75.0 

I 

25.0 

I 

12.5 

I 

13.0 

I 

11.1 

I 

I 

9.4 

I 

3.  1 

I 

-I- 

-I- 

-I 

4 

I 

13 

I 

2 

I 

15 

VERY  USEFUL 

I 

86.7 

I 

13.3 

I 

46.9 

I 

56.5 

I 

22.2 

I 

I 

40.6 

I 

6.3 

I 

*  I  - 

-I- 

-I 

5 

I 

5 

I 

6 

I 

1  1 

ESSENTIAL 

I 

45.5 

I 

54.5 

I 

34.4 

I 

21 .7 

I 

66.7 

I 

I 

15.6 

I 

18.8 

I 

-I  - 

-I- 

-I 

COLUMN 

23 

9 

32 

TOTAL 

71.9 

28. 1 

100.0 

SIGNIFICANCE  * 


1830 


11111  WITH  USER 


8  OUT  OF  10  (  80.0%)  OF  THE  VALIO  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 
MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .281 

RAW  CHI  SQUARE  *  6.22386  WITH  4  OEGREES  OF  FREEDOM. 

CRAMER'S  V  =  .44102 

CONTINGENCY  COEFFICIENT  =  .40352 

LAMBOA  (ASYMMETRIC)  =  .23529  WITH  HELP  DEPENOENT. 

LAMBDA  (SYMMETRIC)  =  .19231 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .08586  WITH  HELP 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .11481 

KENOALL'S  TAU  B  =  .35200.  SIGNIFICANCE  = 

KENOALL'S  TAU  C  =  .35938.  SIGNIFICANCE  = 

GAMMA  =  .62162 

SDMERS'S  0  (ASYMMETRIC)  =  .44444  WITH  HELP 

SOMERS'S  0  (SYMMETRIC)  =  .34264 

ETA  =  .33046  WITH  HELP  OE PENOENT . 

PEARSON'S  R  =  .33046  SIGNIFICANCE  =  .0324 


OE  PENOENT . 


OEPENOENT . 


.  17324  WITH  USER 


.0185 

.0185 

OEPENOENT. 


.44102  WITH  USER 


.27879  WITH  USER 


DEPENOENT. 


DEPENOENT. 


*  *  * 

1  OF  1 


OEPENOENT. 


******************  CROSSTABULATIDN 
STDRE  STORES  SEARCH  STRATEGY  BY  USER 

*********************  +  +  *  +  +  +  +  +  +  +  +  3(?  + 


D  F  *  *  *  * 
USER  TYPE 
*  *  *  *  *  *  * 


USER 

COUNT  I 

RDW  PCT  1 1 NTERMED  END-USER 
COL  PCT  1 1 ARY 


TOT  PCT 
STDRE  _ 

I 

1 

I 

2 

I 

1 

I 

0 

I 

1 

I 

NOT  USEFUL 

I 

.0 

I 

100.0 

I 

I 

.0 

I 

11.1 

I 

I 

.0 

I 

3.  1 

I 

I- 

-I- 

-I 

2 

I 

1 

I 

0 

I 

SDMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I- 

-I- 

-I 

3 

I 

2 

I 

2 

I 

INDIFFERENT 

I 

50.0 

I 

50.0 

I 

I 

8.7 

I 

22.2 

I 

I 

6.3 

I 

6.3 

I 

I- 

-I- 

-I 

4 

I 

5 

I 

2 

I 

VERY  USEFUL 

I 

71.4 

I 

28.6 

I 

I 

21.7 

I 

22.2 

I 

I 

15.6 

I 

6.3 

I 

I- 

-I 

5 

I 

15 

I 

4 

I 

ESSENTIAL 

I 

78.9 

I 

21.1 

I 

I 

65.2 

I 

44.4 

I 

I 

46.9 

I 

12.5 

I 

- 

I- 

-I- 

-I 

COLUMN 

23 

9 

TDTAL 

71.9 

28.  1 

ROW 

TOTAL 


1 

3.  1 


1 

3.  1 


4 

12.5 


7 

2  1.9 


19 

59.4 


32 

100.0 


* 


* 


it  *  * 


PAGE 


7  OUT  DF  10  (  70.0%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0 
MINIMUM  EXPECTED  CELL  FREQUENCY  =  .281 

RAW  CHI  SQUARE  =  4.36453  WITH  4  DEGREES  DF  FREEDDM.  SIGNIFICANCE  =  3589 

CRAMER ' S  V  =  .36931 

CONTINGENCY  COEFFICIENT  =  .34644 

LAMBOA  (ASYMMETRIC)  =  .00000  WITH  STDRE  DEPENDENT.  =  .11111  WITH  USER  DEPENDENT 

LAMBDA  (SYMMETRIC)  =  .04545 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .06351  WITH  STDRE  DEPENDENT  =  11957  WITH  U 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .08296 

KENDALL'S  TAU  B  =  -.21339.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  -.20703.  SIGNIFICANCE  = 

GAMMA  =  -.39850 

SOMERS'S  0  (ASYMMETRIC)  =  -.25604  WITH  STDRE 

SOMERS'S  D  (SYMMETRIC)  =  -.20990 

ETA  =  .26144  WITH  STDRE  DEPENDENT. 

PEARSDN'S  R  =  -.26144  SIGNIFICANCE  =  .0742 


.  104  1 
.  104  1 

DEPENDENT. 

=  .36931  WITH  USER 


.  17785  WITH  USER 
DEPENDENT. 


OEPENDENT . 


*  *  * 

1  DF  1 


DEPENDENT. 


--4 

UJ 

I 


****************** 
FDRMAT  USR  DEF  DUTPUT  FDRMATS 


CRDSSTABULATIDN 

BY  USER 


OF  *************** 
USER  TYPE 


*************************************************  PAGE 


FDRMAT 


INDIFFERENT 


USER 

CDUNT  I 

RDW  PCT  1 1 NTERMED  END-USER 
CDL  PCT  1 1 ARY 

TDT  PCT  I  II  21 

- 1 . _j - j 

2  1  0  1 

100.0  I  .0  I 

8.7  I  .0  I 

6.3  I  .0  I 


VERY  USEFUL 


ESSENTIAL 


I 

I 

I 

I 

-I 

I 

I 

I 

I 

-I 

I 

I 

I 

I 

-I 


RDW 

TDTAL 


2 

6.3 


-I- 

-I 

4 

I 

4 

I 

8 

50.0 

I 

50.0 

I 

25.0 

17.4 

I 

44.4 

I 

12.5 

I 

12.5 

I 

-I- 

-I 

17 

I 

5 

I 

22 

77.3 

I 

22.7 

I 

68.8 

73.9 

I 

55.6 

I 

53.  1 

I 

15.6 

I 

-I- 

-I 

COLUMN 

TDTAL 


23 

71.9 


9 

28. 1 


32 

100.0 


3  DUT  DF  6  (  50.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  *  2.99341  WITH  2  DEGREES  DF  FREEDOM.  SIGNIFICANCE  =  .2239 

CRAMER ' S  V  =  .30585 

CONTINGENCY  COEFFICIENT  =  .29248 

LAMBDA  (ASYMMETRIC)  =  .00000  WITH  FDRMAT  DEPENDENT.  =  .OOOOO  WITH  USER  DEPENDENT 

LAMBDA  (SYMMETRIC)  =  .OOOOO 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .06736  WITH  FDRMAT  DEPENDENT.  =  .08815  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .07636 

.2195 
.2195 


- .  13573. 
- . 11719. 


SIGNIFICANCE  = 
SIGNIFICANCE  = 


KENDALL'S  TAU  B  = 

KENDALL'S  TAU  C  = 

GAMMA  =  -.28302 

SDMERS'S  D  (ASYMMETRIC)  =  -.14493  WITH  FDRMAT 

SDMERS'S  D  (SYMMETRIC)  =  -.13544 

ETA  =  .07246  WITH  FDRMAT  DEPENDENT. 

PEARSDN'S  R  =  -.07246  SIGNIFICANCE  =  .3467 


DEPENDENT. 


.30585  WITH  USER 


-.  12712  WITH  USER 


DEPENDENT . 


DEPENDENT . 


*  *  * 

1  DF  1 


DEPENDENT. 


******************  CRDSSTA8ULATI0N  DF  *************** 
STDFOR  STDRE  USR  OEF  FDRMATS  BY  USER  USER  TYPE 

*************************************************  PAGE 

USER 

COUNT  I 

RDW  PCT  I INTERMED  END-USER  RDW 


CDL  PCT  I I ARY 

TDT  PCT  I  1 

I 

T  — 

2 

I 

TDTAL 

o  i  ururc  l 

1 

- 1 

2  I 

2 

I 

0 

I 

2 

SDMEWHAT  USEFUL  I 

100.0 

I 

.0 

I 

6.3 

I 

8.7 

I 

.0 

I 

I 

6.3 

I 

.0 

I 

-I- 

-I- 

-I 

3  I 

1 

I 

1 

I 

2 

INDIFFERENT  I 

50.0 

I 

50.0 

I 

6.3 

I 

4.3 

I 

11.1 

I 

I 

3. 1 

I 

3.  1 

I 

-I- 

-I- 

-I 

4  I 

1  1 

I 

5 

I 

16 

VERY  USEFUL  I 

68.8 

I 

31.3 

I 

50.0 

I 

47.8 

I 

55.6 

I 

I 

34.4 

I 

15.6 

I 

-I- 

-I- 

-I 

5  I 

9 

I 

3 

I 

12 

ESSENTIAL  I 

75.0 

I 

25.0 

I 

37.5 

I 

39.  1 

I 

33.3 

I 

I 

28.  1 

I 

9.4 

I 

-I- 

CDLUMN 

TDTAL 

23 

71.9 

-I- 

9 

28. 1 

-I 

32 

100.0 

6  OUT  DF  8  (  75.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED 

MINIMUM  EXPECTEO  CELL  FREQUENCY  =  .563 

CELL  FREQUENCY 

LESS 

THAN 

5.0. 

RAW  CHI  SQUARE  =  1.39130  WITH  3  DEGREES  OF  FREEDOM. 

CRAMER'S  V  =  .20851 

CONTINGENCY  COEFFICIENT  =  .20412 

SIGNIFICANCE  = 

.7076 

LAMBDA  (ASYMMETRIC)  =  . OOOOO  WITH  STOFOR  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .00000 

=  .OOOOO 

WITH 

USER 

DEPENDENT. 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .02770  WITH  STDFDR 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .03551 

KENDALL'S  TAU  B  =  -.02376.  SIGNIFICANCE  =  .4449 

DEPENOENT. 

.04947  WITH  USER 

KENDALL'S  TAU  C  *  -.02344.  SIGNIFICANCE  *  .4449 

GAMMA  =  -.04839 

SOMERS'S  0  (ASYMMETRIC)  =  -.02899  WITH  STOFOR  DEPENOENT .  *  -.01948  WITH  USER  OEPENDEN f . 

SOMERS'S  D  (SYMMETRIC)  =  -.02330 

ETA  =  .02689  WITH  STDFDR  DEPENOENT.  =  .20851  WITH  USER  DEPENOENT. 

PEARSDN'S  R  =  .02689  SIGNIFICANCE  =  .4419 


*  *  * 

1  DF  1 


OEPENDENT . 


4 


MENFDR 


MENU  OF  CANNEO 


C  R  0  S  S  T 
FDRMATS 


BULAT 

BY 


I  D  N 
USER 


D  F 


USER  TYPE 
*  *  *  *  *  * 


PAGE 


COUNT 
ROW  PCT 
COL  PCT 
TDT  PCT 


MENFOR 


1 


NOT  USEFUL 


USER 

I 

1 1 NTERMEO  ENO-USER 

I I  ARY 

I  II  2 

I - 1-- . 

1  I  O 

100.0  I  .0 

4.3  I  .0 

3.1  I  .0 


ROW 

TOTAL 


SOMEWHAT  USEFUL 


INOI FFERENT 


1 

I 

0 

I 

100.0 

I 

.0 

I 

4.3 

I 

.0 

I 

3. 1 

I 

.0 

I 

-I- 

-I 

3 

I 

2 

I 

60.0 

I 

40.0 

I 

13.0 

I 

22.2 

I 

9.4 

I 

6.3 

I 

1 

3.  1 


1 

3.  1 


5 

15.6 


VERY  USEFUL 


ESSENTIAL 


~  i  - 

4 

I 

10 

I 

2 

I 

12 

I 

83.3 

I 

16.7 

I 

37.5 

I 

43.5 

I 

22.2 

I 

I 

31 .3 

I 

6.3 

I 

-I- 

-I- 

-I 

5 

I 

8 

I 

5 

I 

13 

I 

61.5 

I 

38.5 

I 

40.6 

I 

34.8 

I 

55.6 

I 

I 

25.0 

I 

15.6 

I 

-  T  - 

COLUMN 

TOTAL 


23 
71  .9 


9 

28.  1 


32 

100.0 


8  OUT  DF  10  (  80.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  *  .281 

RAW  CHI  SQUARE  =  2.59789  WITH  4  OEGREES  OF  FREEOOM .  SIGNIFICANCE  =  .6272 

CRAMER ' S  V  =  .28493 

CONTINGENCY  COEFFICIENT  =  .27402 

LAMBDA  (ASYMMETRIC)  =  .10526  WITH  MENFDR  DEPENDENT.  =  . OOOOO  WITH  USER  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .07143 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .03977  WITH  MENFDR  OEPENDENT .  =  .08304  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .05379 

.2039 
.2039 


. 13906. 
. 14453. 


SIGNIFICANCE 

SIGNIFICANCE 


KENDALL  ' S  TAU  B  « 

KENDALL  7  S  TAU  C  ’» 

GAMMA  =  .26241 

SDMERS  * S  0  (ASYMMETRIC)  =  .17874  WITH  MENFOR 

SDMERS ' S  0  (SYMMETRIC)  =  .13479 

ETA  =  .15296  WITH  MENFOR  OEPENOENT . 

PEARSON ' S  R  =  .15296  SIGNIFICANCE  =  .2016 


OEPENOENT. 


.28493  WITH  USER 


. 10819  WITH  USER 


OEPENDENT. 


OEPENOENT. 


*  *  * 

1  DF  1 


OEPENOENT. 


VO 

I 


******************  CRDSST  ABULAT  IDN  DF  *  *  *  * 
CHARTS  USER  CREATE  CHARTS  BY  USER  USER  TYPE 

*********************  +  +  *  +  ***************** 


USER 

CDUNT  I 

ROW  PCT  I INTERMED  END-USER  RDW 


COL 

PCT 

1 1  ARY 

TDTAL 

TOT 

PCT 

I 

1 

I 

2 

I 

CHARTS  - 

-I  - 

“I- 

-I 

1 

I 

1 

I 

1 

I 

2 

NOT  USEFUL 

I 

50.0 

I 

50.0 

I 

6.3 

I 

4.3 

I 

11.1 

I 

I 

3. 1 

I 

3.  1 

I 

-I- 

-I- 

-I 

2 

I 

5 

I 

1 

I 

6 

SOMEWHAT  USEFUL 

I 

83.3 

I 

16.7 

I 

18.8 

I 

21.7 

I 

11.1 

I 

I 

15.6 

I 

3. 1 

I 

I- 

-I- 

-I 

3 

I 

9 

I 

2 

I 

1  1 

INDIFFERENT 

I 

81.8 

I 

18.2 

I 

34.4 

I 

39.  1 

I 

22.2 

I 

I 

28. 1 

I 

6.3 

I 

I- 

-I- 

-I 

4 

I 

4 

I 

2 

I 

6 

VERY  USEFUL 

I 

66.7 

I 

33.3 

I 

18.8 

I 

17.4 

I 

22.2 

I 

I 

12.5 

I 

6.3 

I 

I- 

-I- 

-I 

5 

I 

4 

I 

3 

I 

7 

ESSENTIAL 

I 

57. 1 

I 

42.9 

I 

21.9 

I 

17.4 

I 

33.3 

I 

I 

12.5 

I 

9.4 

I 

- 

I- 

-I- 

-I 

COLUMN 

23 

9 

32 

TDTAL 

71 .9 

28. 1 

100.0 

*  *  ★ 

PAGE 


8  OUT  OF  10  (  80.0%)  DF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  =  2.23318  WITH  4  DEGREES  DF  FREEDDM.  SIGNIFICANCE  =  .6930 

CRAMER  * S  V  =  .26417 

CONTINGENCY  COEFFICIENT  =  .25541 

LAMBDA  (ASYMMETRIC)  =  .04762  WITH  CHARTS  DEPENDENT.  =  .00000  WITH  USER  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .03333 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .02306  WITH  CHARTS  DEPENDENT.  =  .05825  WITH  USER 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .03305 

KENDALL'S  TAU  B  =  .13039.  SIGNIFICANCE  =  .2115 

KENDALL'S  TAU  C  =  .14453.  SIGNIFICANCE  =  .2115 


GAMMA  =  .22699 

SOMERS'S  D  (ASYMMETRIC)  = 

.  17874  WITH 

CHARTS 

DEPENDENT. 

.09512  WITH  USER 

DEPENDENT 

SOMERS'S  D  (SYMMETRIC)  = 

ETA  =  .  12839  WITH  CHARTS 

.  12416 

DEPENDENT. 

=  .26417  WITH  USER 

DEPENDENT. 

PEARSON'S  R  =  .12839  SIGNIFICANCE  = 

.2419 

*  +  * 

1  DF  1 


DEPENDENT. 


****** 


******* 


*  *  *  * 


D  F 


GRAPHS  USER  CREATE 


************* 


CROSSTABULATION 
GRAPHS  BY  USER 

**************** 


*  *  *  *  * 
USER  TYPE 
******* 


USER 

COUNT  I 

RDW  PCT  I I NTERMED  END-USER  RDW 


COL  PCT 
TOT  PCT 

r.DADUC  _ 

1 1  ARY 

I  1 

I 

2 

I 

TOTAL 

1 

1 

I 

1 

I 

2 

NOT  USEFUL 

50.0 

I 

50.0 

I 

6.3 

4.3 

I 

11.1 

I 

3.  1 

I 

3.  1 

I 

- 

-I- 

-I 

2 

6 

I 

2 

I 

8 

SOMEWHAT  USEFUL 

75.0 

I 

25.0 

I 

25.0 

26.  1 

I 

22.2 

I 

18.8 

I 

6.3 

I 

- 

-I- 

-I 

3 

9 

I 

2 

I 

1  1 

INDIFFERENT 

81.8 

I 

18.2 

I 

34.4 

39.  1 

I 

22.2 

I 

28. 1 

I 

6.3 

I 

- 

-I- 

-I 

4 

3 

I 

1 

I 

4 

VERY  USEFUL 

75.0 

I 

25.0 

I 

12.5 

13.0 

I 

11.1 

I 

9.4 

I 

3.  1 

I 

- 

-I- 

-I 

5 

4 

I 

3 

I 

7 

ESSENTIAL 

57.  1 

I 

42.9 

I 

21.9 

17.4 

I 

33.3 

I 

12.5 

I 

9.4 

I 

- 

-I- 

-I 

CDLUMN 

23 

9 

32 

TOTAL 

71.9 

28.  1 

100.0 

*  *  * 

PAGE 


7  OUT  OF  10  (  70-0%)  OF  THE  VALID  CELLS  HAVE  EXPECTED  CELL  FREQUENCY  LESS  THAN  5.0. 

MINIMUM  EXPECTED  CELL  FREQUENCY  =  .563 

RAW  CHI  SQUARE  =  1.82094  WITH  4  DEGREES  OF  FREEDOM.  SIGNIFICANCE  =  .7686 

CRAMER'S  V  =  .23855 

CONTINGENCY  COEFFICIENT  =  .23204 

LAMBDA  (ASYMMETRIC)  =  .04762  WITH  GRAPHS  DEPENDENT.  =  .00000  WITH  USER  DEPENDENT. 

LAMBDA  (SYMMETRIC)  =  .03333 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .01863  WITH  GRAPHS  DEPENDENT.  =  .04639  WITH  US 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .02658 

KENDALL'S  TAU  B  =  .06022.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  .06641.  SIGNIFICANCE  = 

GAMMA  =  . 10559 

SDMERS'S  D  (ASYMMETRIC)  =  .08213  WITH  GRAPHS 

SOMERS'S  D  (SYMMETRIC)  =  .05743  * 

ETA  =  07537  WITH  GRAPHS  DEPENDENT. 

PEARSDN'S  R  =  .07537  SIGNIFICANCE  =  .3409 


.3561 
.356  1 

DEPENDENT.  *  .04416  WITH  USER  DEPENDENT. 

=  .23855  WITH  USER  DEPENDENT. 


*  *  * 

1  OF  1 


DEPENDENT. 


CO 

KjJ 

I 


****************** 
DDWNLD  DDWNLDAD  INFORMATION 


CRDSSTABULATIDN 

BY  USER 


DF  **************, 
USER  TYPE 


**********  *************  *********.**  **************  *  page 

USER 


CDUNT 
RDW  PCT 
CDL  PCT 
TDT  PCT 


I INTERMED  END-USER 
1 1  ARY 


RDW 

TDTAL 


DWNLD  - 

2 

I 

1 

I 

0 

I 

1 

SDMEWHAT  USEFUL 

I 

100.0 

I 

.0 

I 

3.  1 

I 

4.3 

I 

.0 

I 

I 

3.  1 

I 

.0 

I 

-I 

3 

I 

4 

I 

0 

I 

4 

INDIFFERENT 

I 

100.0 

I 

.0 

I 

12.5 

I 

17.4 

I 

.0 

I 

I 

12.5 

I 

.0 

I 

-I 

4 

I 

6 

I 

2 

I 

8 

VERY  USEFUL 

I 

75.0 

I 

25.0 

I 

25.0 

I 

26.  1 

I 

22.2 

I 

I 

18.8 

I 

6.3 

I 

-I- 

5 

I 

12 

I 

7 

I 

19 

ESSENTIAL 

I 

63.2 

I 

36.8 

I 

59.4 

I 

52.2 

I 

77.8 

I 

I 

37.5 

I 

21.9 

I 

-I- 

CDLUMN 

23 

9 

32 

TDTAL 

71.9 

28.  1 

100.0 

5  DUT  DF  8 

( 

62.5%) 

DF 

THE  VALID  CELLS 

.281 

3  DEGREES  DF  FREEDDM. 


RAW  CHI  SQUARE  =  2.70938  WITH 

CRAMER'S  V  *  .29098 

CONTINGENCY  COEFFICIENT  *  .27939 

LAMBDA  (ASYMMETRIC)  =  .00000  WITH  DDWNLD 

LAMBDA  (SYMMETRIC)  =  .00000 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .06130  WITH  DDWNLD 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .07760 

KENDALL'S  TAU  B  *  .25669.  SIGNIFICANCE  = 

KENDALL'S  TAU  C  =  .24609.  SIGNIFICANCE  = 

GAMMA  =  .56757 

SDMERS'S  D  (ASYMMETRIC)  =  .30435  WITH  DDWNLD 

SDMERS'S  D  (SYMMETRIC)  =  .25301 

ETA  =  .28210  WITH  DDWNLD  DEPENDENT. 

PEARSDN'S  R  =  .28210  SIGNIFICANCE  =  .0589 


SIGNIFICANCE 


.4386 


DEPENDENT.  =  ,00000  WITH  USER  DEPENDENT. 

DEPENDENT.  =  .10569  WITH  USER 

.0667 
.0667 


DEPENDENT. 


.29098  WITH  USER 


.21649  WITH  USER 
DEPENDENT. 


DEPENDENT. 


*  *  * 

1  DF  1 


DEPENDENT. 


*  *  *  *  * 


****** 


****** 


************ 


***** 

REFORM 

***** 


REFORMAT 
*  *  * 


CROSSTABULATION 
INFORMATION  BY  USER 


0  F 


COUNT 
ROW  PCT 
COL  PCT 
TOT  PCT 


REFORM 


1 


NOT  USEFUL 


SOMEWHAT  USEFUL 


INDIFFERENT 


VERY  USEFUL 


ESSENTIAL 


COLUMN 

TOTAL 


8  OUT  OF 


*  *  *  * 

* 

*  *  * 

* 

*************, 

USER 

r 

[INTERMEO  ENO-USER 
[  I  ARY 

ROW 

TOTAL 

1 

I 

2 

I 

0 

I 

1 

I 

1 

.0 

I 

100.0 

I 

3.  1 

.0 

I 

11.1 

I 

.0 

I 

3.  1 

I 

-I- 

-I 

2 

I 

1 

I 

3 

66.7 

I 

33.3 

I 

9.4 

8.7 

I 

11.1 

I 

6.3 

I 

3.  1 

I 

-I- 

-I 

4 

I 

0 

I 

4 

100.0 

I 

.0 

I 

12.5 

17.4 

I 

.0 

I 

12.5 

I 

.0 

I 

-I- 

“I 

7 

I 

1 

I 

8 

87.5 

I 

12.5 

I 

25.0 

30.4 

I 

11.1 

I 

21.9 

I 

3.  1 

I 

•  I  - 

-I 

10 

I 

6 

I 

16 

62.5 

I 

37.5 

I 

50.0 

43.5 

I 

66.7 

I 

31.3 

I 

18.8 

I 

23 

71.9 

I- 

9 

28.  1 

-I 

32 

100.0 

80.0%) 

OF 

THE  VALIO  CELLS  HAVE  EXPECTEO  CELL 

USER  TYPE 
****** 


PAGE 


.281 

4  OEGREES  OF  FREEOOM . 


RAW  CHI  SQUARE  =  5.82287  WITH 

CRAMER'S  V  =  .42657 

CONTINGENCY  COEFFICIENT  =  .39237 

LAMBOA  (ASYMMETRIC)  =  .00000  WITH  REFORM  OEPENOENT 

LAMBOA  (SYMMETRIC)  =  .04000 

UNCERTAINTY  COEFFICIENT  (ASYMMETRIC)  =  .08531  WITH  REFORM 

UNCERTAINTY  COEFFICIENT  (SYMMETRIC)  =  .11663 


SIGNIFICANCE  =  .2128 


.11111  WITH  USER 


OEPENOENT. 


OEPENOENT. 

. 18427  WITH  USER 


SIGNIFICANCE  = 
SIGNIFICANCE  = 


KENOALL'S  TAU  B  =  .11325. 

KENOALL'S  TAU  C  =  .11719. 

GAMMA  =  .21739 

SOMERS'S  0  (ASYMMETRIC)  =  .14493  WITH  REFORM 

SOMERS'S  0  (SYMMETRIC)  =  .10989 

ETA  =  .00963  WITH  REFORM  OEPENOENT. 

PEARSON'S  R  =  .00963  SIGNIFICANCE  =  .4791 


.2486 
.  2486 

OEPENOENT. 


.42657  WITH  USER 


.08850  WITH  USER 
OEPENOENT . 


OEPENOENT. 
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